[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:33 dist-test-slave-mfx9 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:33 dist-test-slave-mfx9 krb5kdc[399](info): set up 2 sockets
Sep 27 02:52:33 dist-test-slave-mfx9 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 02:52:34.363479 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:52:34 dist-test-slave-mfx9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:52:34.471047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:34.548825 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:34.549077 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:34.549108 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:34.550388 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:34.550431 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:34.550444 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:34.550457 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:34.550467 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:34.551888 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:34.552073 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:34.552263 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:34.556905 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:34.558341 412 server_base.cc:1034] running on GCE node
W20260927 02:52:34.558450 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:34.558475 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:34.558732 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:34.559044 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:34.560184 412 hybrid_clock.cc:648] HybridClock initialized: now 1790477554560158 us; error 39 us; skew 500 ppm
Sep 27 02:52:34 dist-test-slave-mfx9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:34.562456 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:34.562970 412 webserver.cc:460] Webserver started at http://127.0.98.62:42957/ using document root <none> and password file <none>
I20260927 02:52:34.563161 412 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:34.563206 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:34.563328 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:34.564373 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "f4750a7e31d742e892040dba10c7ba33"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "ac59c8b2f1115732816306a1049cda4a"
server_key_iv: "7d1888aba1ae1a7581ce56a7a200f990"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.564718 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4750a7e31d742e892040dba10c7ba33"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "ac59c8b2f1115732816306a1049cda4a"
server_key_iv: "7d1888aba1ae1a7581ce56a7a200f990"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.565948 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:34.566536 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:34.566682 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:34.566829 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "f4750a7e31d742e892040dba10c7ba33"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "ac59c8b2f1115732816306a1049cda4a"
server_key_iv: "7d1888aba1ae1a7581ce56a7a200f990"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.566891 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:34.577860 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:34.579957 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:34.580096 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:34.583745 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36307
I20260927 02:52:34.583776 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36307 every 8 connection(s)
I20260927 02:52:34.584107 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:34.584965 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:34.585274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 412
I20260927 02:52:34.585378 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:34.586122 392 external_mini_cluster.cc:1428] Setting key 8673e298db3b7d18ab492c8b2eb6f060
I20260927 02:52:34.587805 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33: Bootstrap starting.
I20260927 02:52:34.588330 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:34.588573 480 log.cc:824] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:34.589246 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33: No bootstrap required, opened a new log
I20260927 02:52:34.590790 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4750a7e31d742e892040dba10c7ba33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260927 02:52:34.590935 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:34.590958 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4750a7e31d742e892040dba10c7ba33, State: Initialized, Role: FOLLOWER
I20260927 02:52:34.591076 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4750a7e31d742e892040dba10c7ba33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260927 02:52:34.591138 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:34.591159 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:34.591203 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:34.591711 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4750a7e31d742e892040dba10c7ba33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260927 02:52:34.591794 480 leader_election.cc:302] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4750a7e31d742e892040dba10c7ba33; no voters:
I20260927 02:52:34.591943 480 leader_election.cc:288] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:34.591988 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:34.592094 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [term 1 LEADER]: Becoming Leader. State: Replica: f4750a7e31d742e892040dba10c7ba33, State: Running, Role: LEADER
I20260927 02:52:34.592178 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4750a7e31d742e892040dba10c7ba33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260927 02:52:34.592202 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:34.592492 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4750a7e31d742e892040dba10c7ba33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4750a7e31d742e892040dba10c7ba33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } } }
I20260927 02:52:34.592490 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4750a7e31d742e892040dba10c7ba33. Latest consensus state: current_term: 1 leader_uuid: "f4750a7e31d742e892040dba10c7ba33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4750a7e31d742e892040dba10c7ba33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } } }
I20260927 02:52:34.592684 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:34.592650 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:34.592993 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:34.593461 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:34.595178 492 catalog_manager.cc:1347] Generated new cluster ID: f037a0c6caa44149b2e4d4890a7c9fa9
I20260927 02:52:34.595227 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:52:34 dist-test-slave-mfx9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:34.603132 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:34.587104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.0.98.62:36307)
0927 02:52:34.587289 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:34.587298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:34.587460 (+ 162us) server_negotiation.cc:407] Connection header received
0927 02:52:34.589580 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:34.589588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:34.589646 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:34.589806 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:34.591296 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.592718 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.594137 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.594710 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.594838 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:34.598815 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:34.598819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:34.598819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:34.598830 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:34.601231 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:34.601709 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:34.601711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:34.601711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:34.601810 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:34.602062 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:34.602063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:34.602063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:34.602331 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:34.602472 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:34.602743 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 02:52:34.602917 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260927 02:52:34.611817 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:34.612339 492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:34.614961 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4750a7e31d742e892040dba10c7ba33: Generated new TSK 0
I20260927 02:52:34.615108 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:52:34.646013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:34.742838 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:34.743023 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:34.743044 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:34.744143 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:34.744182 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:34.744207 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:34.745599 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:34.745811 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:34.746024 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:34.750885 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:34.750893 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:34.750897 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:34.751099 506 server_base.cc:1034] running on GCE node
I20260927 02:52:34.751346 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:34.751546 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:34.752696 506 hybrid_clock.cc:648] HybridClock initialized: now 1790477554752681 us; error 30 us; skew 500 ppm
Sep 27 02:52:34 dist-test-slave-mfx9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:34.755192 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:52:34.755640 506 webserver.cc:460] Webserver started at http://127.0.98.1:40321/ using document root <none> and password file <none>
I20260927 02:52:34.755839 506 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:34.755885 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:34.756006 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:34.756997 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "cbfa95abf85049dca16a97e96492fe4e"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "9288c8f837a7caf398a07b6479522a18"
server_key_iv: "612ca74a1f5652ec6fcb1735ba1a6820"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.757325 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cbfa95abf85049dca16a97e96492fe4e"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "9288c8f837a7caf398a07b6479522a18"
server_key_iv: "612ca74a1f5652ec6fcb1735ba1a6820"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.758517 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:52:34.759184 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:34.759325 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:34.759399 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "cbfa95abf85049dca16a97e96492fe4e"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "9288c8f837a7caf398a07b6479522a18"
server_key_iv: "612ca74a1f5652ec6fcb1735ba1a6820"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.759462 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:34.775625 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:34.777282 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:34.777407 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:34.777598 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:34.777863 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:34.777894 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:34.777921 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:34.777938 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:34.783156 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34005
I20260927 02:52:34.783215 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34005 every 8 connection(s)
I20260927 02:52:34.783473 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:52:34 dist-test-slave-mfx9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:34.791038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 506
I20260927 02:52:34.791132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:34.791292 392 external_mini_cluster.cc:1428] Setting key b8a2e2d21d8de0d9b28a514e53780032
I20260927 02:52:34.792114 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:34.784202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49007 (local address 127.0.98.62:36307)
0927 02:52:34.784276 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:34.784278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:34.784431 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:52:34.784925 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:34.784927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:34.784939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:34.784985 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:34.785874 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.786374 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.787384 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.787582 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.787638 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:34.789630 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:34.789631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:34.789632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:34.789638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:34.790704 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:34.791163 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:34.791164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:34.791165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:34.791225 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:34.791495 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:34.791495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:34.791496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:34.791617 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:34.791671 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:34.791939 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 02:52:34.792027 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:52:34.792330 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:34.784162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.1:49007)
0927 02:52:34.784312 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:34.784355 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:34.784832 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:34.785065 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:34.785069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:34.785598 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:34.785777 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:34.785781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.786455 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.786457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:34.787296 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:34.787298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.787651 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.787652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:34.787767 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:34.788178 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:34.788185 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:34.789540 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:34.790808 (+ 1268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:34.790814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:34.790816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:34.791093 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:34.791291 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:34.791292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:34.791292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:34.791421 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:34.791674 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:34.791681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:34.791813 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 02:52:34.791912 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:52:34.792056 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260927 02:52:34.792826 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36307
I20260927 02:52:34.792919 635 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:34.793095 635 heartbeater.cc:511] Master 127.0.98.62:36307 requested a full tablet report, sending...
I20260927 02:52:34.793670 444 ts_manager.cc:194] Registered new tserver with Master: cbfa95abf85049dca16a97e96492fe4e (127.0.98.1:34005)
I20260927 02:52:34.794404 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49007
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:52:34.812721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:34.889540 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:34.889739 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:34.889775 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:34.890918 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:34.890959 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:34.890991 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:34.892366 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:34.892614 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:34.892881 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:34.897699 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:34.897683 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:34.897710 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:34.897917 640 server_base.cc:1034] running on GCE node
I20260927 02:52:34.898114 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:34.898340 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:34.899478 640 hybrid_clock.cc:648] HybridClock initialized: now 1790477554899465 us; error 31 us; skew 500 ppm
Sep 27 02:52:34 dist-test-slave-mfx9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:34.901886 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:52:34.902371 640 webserver.cc:460] Webserver started at http://127.0.98.2:36597/ using document root <none> and password file <none>
I20260927 02:52:34.902563 640 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:34.902606 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:34.902730 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:34.903764 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "e9a9307808d54133a77b54c0b6f69075"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "61f57cc3d602755dbce62d2e68ae6024"
server_key_iv: "e6d4d4d23cc3b71835e4d5dd965a6e66"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.904059 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e9a9307808d54133a77b54c0b6f69075"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "61f57cc3d602755dbce62d2e68ae6024"
server_key_iv: "e6d4d4d23cc3b71835e4d5dd965a6e66"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.905277 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:52:34.905936 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:34.906100 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:34.906185 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "e9a9307808d54133a77b54c0b6f69075"
format_stamp: "Formatted at 2026-09-27 02:52:34 on dist-test-slave-mfx9"
server_key: "61f57cc3d602755dbce62d2e68ae6024"
server_key_iv: "e6d4d4d23cc3b71835e4d5dd965a6e66"
server_key_version: "encryptionkey@0"
I20260927 02:52:34.906242 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:34.930424 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:34.932233 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:34.932366 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:34.932574 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:34.932862 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:34.932895 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:34.932922 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:34.932945 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:34.938551 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36627
I20260927 02:52:34.938616 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36627 every 8 connection(s)
I20260927 02:52:34.938933 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:52:34 dist-test-slave-mfx9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:34.946729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 640
I20260927 02:52:34.946847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:34.947009 392 external_mini_cluster.cc:1428] Setting key 4bdf56e9fc285f7796cc070442844a0e
I20260927 02:52:34.947206 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:34.939635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44599 (local address 127.0.98.62:36307)
0927 02:52:34.939695 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:34.939697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:34.939871 (+ 174us) server_negotiation.cc:407] Connection header received
0927 02:52:34.940331 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:34.940332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:34.940341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:34.940401 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:34.941250 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.941725 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.942647 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.942888 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.942935 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:34.944887 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:34.944888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:34.944888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:34.944895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:34.945988 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:34.946321 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:34.946322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:34.946322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:34.946365 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:34.946539 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:34.946540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:34.946540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:34.946664 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:34.946733 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:34.947025 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 02:52:34.947107 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:52:34.947355 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:34.939600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.2:44599)
0927 02:52:34.939751 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:34.939794 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:34.940259 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:34.940465 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:34.940469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:34.940998 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:34.941184 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:34.941188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.941788 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.941788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:34.942590 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:34.942591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:34.942948 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:34.942949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:34.943041 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:34.943436 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:34.943443 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:34.944770 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:34.946055 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:34.946060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:34.946061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:34.946267 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:34.946409 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:34.946409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:34.946409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:34.946493 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:34.946720 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:34.946722 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:34.946886 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 02:52:34.946984 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:52:34.947121 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260927 02:52:34.947824 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36307
I20260927 02:52:34.947912 769 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:34.948073 769 heartbeater.cc:511] Master 127.0.98.62:36307 requested a full tablet report, sending...
I20260927 02:52:34.948426 444 ts_manager.cc:194] Registered new tserver with Master: e9a9307808d54133a77b54c0b6f69075 (127.0.98.2:36627)
I20260927 02:52:34.948879 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44599
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:52:34.969818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:35.052551 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:35.052750 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:35.052772 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:35.053850 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:35.053884 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:35.053908 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:35.055318 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:35.055506 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:35.055701 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:35.060312 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:35.060403 774 server_base.cc:1034] running on GCE node
W20260927 02:52:35.060321 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:35.060321 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:35.060772 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:35.060964 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:35.062071 774 hybrid_clock.cc:648] HybridClock initialized: now 1790477555062061 us; error 26 us; skew 500 ppm
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:35.064411 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:52:35.064886 774 webserver.cc:460] Webserver started at http://127.0.98.3:40919/ using document root <none> and password file <none>
I20260927 02:52:35.065099 774 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:35.065143 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:35.065264 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:35.066305 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "6dd23749188a48cab207e0f929c00701"
format_stamp: "Formatted at 2026-09-27 02:52:35 on dist-test-slave-mfx9"
server_key: "9e66bb5cbf551c6f05f7f92023c8a99a"
server_key_iv: "5aaa964c84c1d785e6a7519c0f51d52c"
server_key_version: "encryptionkey@0"
I20260927 02:52:35.066627 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6dd23749188a48cab207e0f929c00701"
format_stamp: "Formatted at 2026-09-27 02:52:35 on dist-test-slave-mfx9"
server_key: "9e66bb5cbf551c6f05f7f92023c8a99a"
server_key_iv: "5aaa964c84c1d785e6a7519c0f51d52c"
server_key_version: "encryptionkey@0"
I20260927 02:52:35.068209 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:35.068863 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:35.069006 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:35.069082 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "6dd23749188a48cab207e0f929c00701"
format_stamp: "Formatted at 2026-09-27 02:52:35 on dist-test-slave-mfx9"
server_key: "9e66bb5cbf551c6f05f7f92023c8a99a"
server_key_iv: "5aaa964c84c1d785e6a7519c0f51d52c"
server_key_version: "encryptionkey@0"
I20260927 02:52:35.069146 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:35.085914 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:35.087924 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:35.088053 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:35.088266 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:35.088548 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:35.088584 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:35.088613 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:35.088635 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:35.095068 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44075
I20260927 02:52:35.095125 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44075 every 8 connection(s)
I20260927 02:52:35.095419 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:35.103828 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:35.096075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51143 (local address 127.0.98.62:36307)
0927 02:52:35.096142 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:35.096144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:35.096312 (+ 168us) server_negotiation.cc:407] Connection header received
0927 02:52:35.096786 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:35.096787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:35.096797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:35.096870 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:35.097749 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:35.098264 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:35.099277 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:35.099457 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:35.099507 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:35.101473 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:35.101474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:35.101474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:35.101481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:35.102588 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:35.103020 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:35.103021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:35.103021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:35.103079 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:35.103275 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:35.103276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:35.103276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:35.103412 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:35.103484 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:35.103693 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 02:52:35.103760 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:52:35.103984 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:35.096063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.3:51143)
0927 02:52:35.096201 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:35.096236 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:35.096685 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:35.096936 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:35.096939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:35.097492 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:35.097682 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:35.097686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:35.098327 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:35.098327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:35.099206 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:35.099209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:35.099515 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:35.099516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:35.099617 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:35.100088 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:35.100097 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:35.101395 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:35.102679 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:35.102684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:35.102686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:35.102936 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:35.103136 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:35.103136 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:35.103136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:35.103224 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:35.103470 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:35.103472 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:35.103579 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 02:52:35.103653 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:52:35.103766 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260927 02:52:35.104456 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36307
I20260927 02:52:35.104513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 774
I20260927 02:52:35.104540 903 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:35.104588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:35.104708 903 heartbeater.cc:511] Master 127.0.98.62:36307 requested a full tablet report, sending...
I20260927 02:52:35.104737 392 external_mini_cluster.cc:1428] Setting key b44c9176957f36452fddd30a09e283b0
I20260927 02:52:35.105089 444 ts_manager.cc:194] Registered new tserver with Master: 6dd23749188a48cab207e0f929c00701 (127.0.98.3:44075)
I20260927 02:52:35.105458 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51143
I20260927 02:52:35.105669 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:52:35.126593 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:35.119548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.0.98.1:34005)
0927 02:52:35.119711 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:35.119716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:35.119726 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:52:35.119751 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:35.119753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:35.119766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:35.119827 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:35.120307 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:35.121016 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:35.121760 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:35.122058 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:35.122105 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:35.124246 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:35.124254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:35.124255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:35.124261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:35.125639 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:35.125951 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:35.125954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:35.125955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:35.126024 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:35.126167 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:35.126167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:35.126167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:35.126308 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:35.126418 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:35.126453 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 02:52:35.126515 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
W20260927 02:52:35.126955 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44912
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:52:35.147338 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:35.140855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.0.98.1:34005)
0927 02:52:35.140926 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:35.140927 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:35.141039 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:52:35.141109 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:35.141110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:35.141123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:35.141169 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:35.141688 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:35.142185 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:35.142936 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:35.143133 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:35.143185 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:35.145203 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:35.145205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:35.145205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:35.145215 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:35.146376 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:35.146654 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:35.146655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:35.146656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:35.146702 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:35.146895 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:35.146895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:35.146896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:35.146985 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:35.147086 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:35.147165 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:52:35.147247 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:52:35.147890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 506
I20260927 02:52:35.151952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 640
I20260927 02:52:35.156735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 774
I20260927 02:52:35.160851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 412
2026-09-27T02:52:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1750 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[917](info): set up 2 sockets
Sep 27 02:52:35 dist-test-slave-mfx9 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 02:52:36.114012 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:52:36.219430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33541
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:36.295547 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:36.295744 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:36.295781 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:36.296900 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:36.296940 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:36.296957 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:36.296973 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:36.296989 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:36.298410 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:36.298641 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:36.298861 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:36.304543 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:36.304875 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:36.304971 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:36.305042 930 server_base.cc:1034] running on GCE node
I20260927 02:52:36.305320 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:36.305591 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:36.306725 930 hybrid_clock.cc:648] HybridClock initialized: now 1790477556306702 us; error 34 us; skew 500 ppm
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:36.309026 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:36.309453 930 webserver.cc:460] Webserver started at http://127.0.98.62:46795/ using document root <none> and password file <none>
I20260927 02:52:36.309623 930 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:36.309659 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:36.309744 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:36.310734 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "13461556db5d457786b00aa8cb761bdb"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "e99d18b35ef0c63d2de853e4e3f98e01"
server_key_iv: "b967d1780597ee57cfb77367bc52e397"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.311070 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "13461556db5d457786b00aa8cb761bdb"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "e99d18b35ef0c63d2de853e4e3f98e01"
server_key_iv: "b967d1780597ee57cfb77367bc52e397"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.312709 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:36.313380 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.313510 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:36.313583 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "13461556db5d457786b00aa8cb761bdb"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "e99d18b35ef0c63d2de853e4e3f98e01"
server_key_iv: "b967d1780597ee57cfb77367bc52e397"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.313644 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:36.348735 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:36.350497 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:36.350618 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:36.354055 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33541
I20260927 02:52:36.354101 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33541 every 8 connection(s)
I20260927 02:52:36.354393 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:36.354974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 930
I20260927 02:52:36.355046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:36.355177 392 external_mini_cluster.cc:1428] Setting key c3b7329974daec1707c279cec9d3a42b
I20260927 02:52:36.355252 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:36.358057 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb: Bootstrap starting.
I20260927 02:52:36.358691 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:36.359001 998 log.cc:824] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:36.359642 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb: No bootstrap required, opened a new log
I20260927 02:52:36.361514 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13461556db5d457786b00aa8cb761bdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } }
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:36.361681 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:36.361713 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13461556db5d457786b00aa8cb761bdb, State: Initialized, Role: FOLLOWER
I20260927 02:52:36.361838 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13461556db5d457786b00aa8cb761bdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } }
I20260927 02:52:36.361922 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:36.361959 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:36.362005 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:36.362516 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13461556db5d457786b00aa8cb761bdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } }
I20260927 02:52:36.362609 998 leader_election.cc:302] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13461556db5d457786b00aa8cb761bdb; no voters:
I20260927 02:52:36.362749 998 leader_election.cc:288] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:36.362807 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:36.362900 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [term 1 LEADER]: Becoming Leader. State: Replica: 13461556db5d457786b00aa8cb761bdb, State: Running, Role: LEADER
I20260927 02:52:36.362977 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13461556db5d457786b00aa8cb761bdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } }
I20260927 02:52:36.363018 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:36.363258 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13461556db5d457786b00aa8cb761bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13461556db5d457786b00aa8cb761bdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } }
I20260927 02:52:36.363320 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:36.363427 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13461556db5d457786b00aa8cb761bdb. Latest consensus state: current_term: 1 leader_uuid: "13461556db5d457786b00aa8cb761bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13461556db5d457786b00aa8cb761bdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33541 } } }
I20260927 02:52:36.363469 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:36.363942 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:36.364413 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:36.365686 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.355714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.0.98.62:33541)
0927 02:52:36.355895 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:36.355903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:36.355926 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:52:36.356527 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:36.356534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:36.356580 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:36.356688 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:36.357372 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.358284 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.359305 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.359752 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.359859 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.362176 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:36.362180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:36.362180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:36.362189 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:36.364086 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.364449 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.364453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.364453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.364555 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.364750 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.364751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.364752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.365050 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:36.365184 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:36.365416 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 02:52:36.365552 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":57,"threads_started":1}
I20260927 02:52:36.365888 1008 catalog_manager.cc:1347] Generated new cluster ID: 2246a74e657342f4a321dc0f754de337
I20260927 02:52:36.365937 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:36.384406 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:36.384940 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:36.390002 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13461556db5d457786b00aa8cb761bdb: Generated new TSK 0
I20260927 02:52:36.390157 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:52:36.425294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33541
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:36.505178 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:36.505368 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:36.505391 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:36.506479 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:36.506513 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:36.506541 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:36.507949 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:36.508127 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:36.508301 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:52:36.513108 1024 server_base.cc:1034] running on GCE node
W20260927 02:52:36.513042 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:36.513023 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:36.513042 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:36.513471 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:36.513695 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:36.514828 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790477556514807 us; error 34 us; skew 500 ppm
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:36.517134 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:52:36.517541 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35713/ using document root <none> and password file <none>
I20260927 02:52:36.517716 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:36.517753 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:36.517848 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:36.518831 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "cb5719f6d657417b92b26c2b3c9a3f0d"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "85e75ca16b06acd62499a624d59536ed"
server_key_iv: "07f12cc51910981f83a3b88f3782c0c5"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.519137 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cb5719f6d657417b92b26c2b3c9a3f0d"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "85e75ca16b06acd62499a624d59536ed"
server_key_iv: "07f12cc51910981f83a3b88f3782c0c5"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.520238 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:36.520849 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.521003 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:36.521068 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "cb5719f6d657417b92b26c2b3c9a3f0d"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "85e75ca16b06acd62499a624d59536ed"
server_key_iv: "07f12cc51910981f83a3b88f3782c0c5"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.521116 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:36.531658 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:36.533257 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:36.533365 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:36.533521 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:36.533779 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:36.533829 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.533849 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:36.533865 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.539040 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35413
I20260927 02:52:36.539103 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35413 every 8 connection(s)
I20260927 02:52:36.539355 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:52:36.539507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 1024
I20260927 02:52:36.539597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:36.539763 392 external_mini_cluster.cc:1428] Setting key afcd768b412c86fc0eb38c0effbf1cc7
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:52:36.548357 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.540080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57501 (local address 127.0.98.62:33541)
0927 02:52:36.540120 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:36.540122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:36.540339 (+ 217us) server_negotiation.cc:407] Connection header received
0927 02:52:36.540895 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:36.540896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:36.540907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:36.540971 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:36.541972 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.542463 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.543496 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.543713 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.543770 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.545913 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:36.545914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:36.545915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:36.545922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:36.547183 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.547520 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.547521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.547521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.547573 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.547778 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.547778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.547779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.547884 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:36.547939 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:36.548187 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:52:36.548267 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:52:36.548511 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.540077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33541 (local address 127.0.98.1:57501)
0927 02:52:36.540222 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:36.540266 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:36.540797 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:36.541025 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:36.541028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:36.541623 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:36.541885 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.541891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.542548 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.542549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.543415 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.543417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.543782 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.543782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.543917 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.544369 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:36.544377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:36.545829 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:36.547252 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:36.547259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:36.547261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:36.547457 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:36.547623 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:36.547624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:36.547624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:36.547725 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:36.547935 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:36.547941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:36.548071 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:52:36.548164 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:52:36.548309 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260927 02:52:36.548969 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33541
I20260927 02:52:36.549050 1153 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:52:36.549228 1153 heartbeater.cc:511] Master 127.0.98.62:33541 requested a full tablet report, sending...
I20260927 02:52:36.549718 962 ts_manager.cc:194] Registered new tserver with Master: cb5719f6d657417b92b26c2b3c9a3f0d (127.0.98.1:35413)
I20260927 02:52:36.550403 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:52:36.565117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33541
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:36.642872 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:36.643047 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:36.643069 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:36.644152 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:36.644186 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:36.644209 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:36.645525 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:36.645704 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:36.645869 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:36.650365 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:36.650410 1158 server_base.cc:1034] running on GCE node
W20260927 02:52:36.650404 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:36.650372 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:36.650715 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:36.650961 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:36.652091 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790477556652079 us; error 28 us; skew 500 ppm
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:36.654451 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:52:36.654950 1158 webserver.cc:460] Webserver started at http://127.0.98.2:34133/ using document root <none> and password file <none>
I20260927 02:52:36.655162 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:36.655210 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:36.655335 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:36.656340 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "482bceab887f4444bc4f8c356945890d"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "5b659058d78d33b7380073b2ff8b2442"
server_key_iv: "0d65c364fdd1961b698427754beb4569"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.656651 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "482bceab887f4444bc4f8c356945890d"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "5b659058d78d33b7380073b2ff8b2442"
server_key_iv: "0d65c364fdd1961b698427754beb4569"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.657847 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:52:36.658473 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.658631 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:36.658707 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "482bceab887f4444bc4f8c356945890d"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "5b659058d78d33b7380073b2ff8b2442"
server_key_iv: "0d65c364fdd1961b698427754beb4569"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.658798 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:36.673651 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:36.675475 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:36.675609 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:36.675804 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:36.676075 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:36.676107 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.676134 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:36.676151 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.681362 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45213
I20260927 02:52:36.681406 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45213 every 8 connection(s)
I20260927 02:52:36.681677 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:36.689368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 1158
I20260927 02:52:36.689461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:36.689625 392 external_mini_cluster.cc:1428] Setting key 714fba72fda7199d122a5998d5a10e68
I20260927 02:52:36.689880 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.682345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52777 (local address 127.0.98.62:33541)
0927 02:52:36.682393 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:36.682395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:36.682592 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:52:36.683084 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:36.683086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:36.683095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:36.683142 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:36.683945 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.684448 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.685442 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.685622 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.685666 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.687598 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:36.687599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:36.687599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:36.687605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:36.688648 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.688982 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.688982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.688983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.689032 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.689249 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.689251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.689251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.689397 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:36.689479 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:36.689732 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 02:52:36.689813 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:52:36.690066 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.682341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33541 (local address 127.0.98.2:52777)
0927 02:52:36.682485 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:36.682518 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:36.683006 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:36.683194 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:36.683197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:36.683700 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:36.683883 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.683887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.684557 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.684558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.685375 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.685377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.685731 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.685732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.685836 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.686277 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:36.686288 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:36.687520 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:36.688707 (+ 1187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:36.688712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:36.688713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:36.688906 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:36.689074 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:36.689074 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:36.689075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:36.689164 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:36.689468 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:36.689472 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:36.689602 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:52:36.689699 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:52:36.689847 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260927 02:52:36.690455 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33541
I20260927 02:52:36.690532 1287 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:36.690688 1287 heartbeater.cc:511] Master 127.0.98.62:33541 requested a full tablet report, sending...
I20260927 02:52:36.691049 962 ts_manager.cc:194] Registered new tserver with Master: 482bceab887f4444bc4f8c356945890d (127.0.98.2:45213)
I20260927 02:52:36.691493 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52777
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:52:36.711673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33541
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:36.789451 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:36.789639 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:36.789660 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:36.790733 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:36.790782 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:36.790808 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:36.792145 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:36.792316 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:36.792477 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:36.796975 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:36.796974 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:36.797133 1292 server_base.cc:1034] running on GCE node
W20260927 02:52:36.797147 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:36.797376 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:36.797583 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:36.798732 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790477556798717 us; error 27 us; skew 500 ppm
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:36.800940 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:52:36.801391 1292 webserver.cc:460] Webserver started at http://127.0.98.3:44519/ using document root <none> and password file <none>
I20260927 02:52:36.801558 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:36.801594 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:36.801682 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:36.802603 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c81b1914ed947dc895b5dff8704a287"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "3b3455e9d96fc3e1150871c66f3af5b7"
server_key_iv: "a01067b61588ee3d0d993fc9cd3858ce"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.802915 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c81b1914ed947dc895b5dff8704a287"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "3b3455e9d96fc3e1150871c66f3af5b7"
server_key_iv: "a01067b61588ee3d0d993fc9cd3858ce"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.804020 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:36.804630 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.804809 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:36.804886 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "9c81b1914ed947dc895b5dff8704a287"
format_stamp: "Formatted at 2026-09-27 02:52:36 on dist-test-slave-mfx9"
server_key: "3b3455e9d96fc3e1150871c66f3af5b7"
server_key_iv: "a01067b61588ee3d0d993fc9cd3858ce"
server_key_version: "encryptionkey@0"
I20260927 02:52:36.804953 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:36.824765 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:36.826789 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:36.826910 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:36.827076 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:36.827322 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:36.827350 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.827370 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:36.827386 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:36.832502 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35589
I20260927 02:52:36.832577 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35589 every 8 connection(s)
I20260927 02:52:36.832854 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:52:36.835737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 1292
I20260927 02:52:36.835824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:36.835953 392 external_mini_cluster.cc:1428] Setting key 111e7fc3f345e9cb3f225bec4510df9d
Sep 27 02:52:36 dist-test-slave-mfx9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:36.841235 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.833528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37485 (local address 127.0.98.62:33541)
0927 02:52:36.833578 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:36.833581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:36.833755 (+ 174us) server_negotiation.cc:407] Connection header received
0927 02:52:36.834229 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:36.834231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:36.834242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:36.834304 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:36.835246 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.835738 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.836925 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.837119 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.837172 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.839032 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:36.839033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:36.839034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:36.839041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:36.840042 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.840393 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.840394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.840394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.840445 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.840651 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.840652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.840653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.840768 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:36.840823 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:36.841065 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 02:52:36.841147 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:52:36.841378 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.833506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33541 (local address 127.0.98.3:37485)
0927 02:52:36.833645 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:36.833681 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:36.834142 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:36.834372 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:36.834375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:36.834963 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:36.835163 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.835168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.835823 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.835824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.836855 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.836857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.837183 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.837184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.837317 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.837686 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:36.837693 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:36.838954 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:36.840110 (+ 1156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:36.840116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:36.840118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:36.840317 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:36.840495 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:36.840496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:36.840496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:36.840597 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:36.840822 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:36.840828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:36.840952 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 02:52:36.841035 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:52:36.841172 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260927 02:52:36.841784 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33541
I20260927 02:52:36.841856 1421 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:36.841984 1421 heartbeater.cc:511] Master 127.0.98.62:33541 requested a full tablet report, sending...
I20260927 02:52:36.842279 962 ts_manager.cc:194] Registered new tserver with Master: 9c81b1914ed947dc895b5dff8704a287 (127.0.98.3:35589)
I20260927 02:52:36.842626 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37485
I20260927 02:52:36.847378 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:52:36.853878 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.848790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.0.98.62:33541)
0927 02:52:36.848821 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:36.848823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:36.848948 (+ 125us) server_negotiation.cc:407] Connection header received
0927 02:52:36.849003 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:36.849004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:36.849013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:36.849062 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:36.849560 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.850014 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.850692 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.850943 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.850990 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.851937 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:36.851939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:36.851939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:36.851945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:36.852913 (+ 968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.853188 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.853189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.853189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.853238 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:36.853411 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:36.853412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:36.853412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:36.853542 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:36.853622 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:36.853723 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:52:36.853797 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 02:52:36.856052 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:52:36.857513 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:36.863507 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.859667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35589 (local address 127.0.0.1:48412)
0927 02:52:36.859772 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:36.859779 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:36.859814 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:36.860127 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:36.860127 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:36.860129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:36.860158 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.860159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.861520 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.861521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.862352 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.862355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.863228 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.863229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.863302 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.863315 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:52:36.863388 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:52:36.863436 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260927 02:52:36.863512 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.859845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.0.98.3:35589)
0927 02:52:36.859984 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:36.859988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:36.859996 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:36.860022 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:36.860023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:36.860036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:36.860222 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:36.860278 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.861416 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.862431 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.863149 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.863219 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.863272 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:36.863405 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:52:36.863442 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260927 02:52:36.863893 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.859443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45213 (local address 127.0.0.1:33004)
0927 02:52:36.859599 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:36.859635 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:36.859690 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:36.860072 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:36.860073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:36.860075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:36.860135 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.860137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.861501 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.861502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.862749 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.862751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.863572 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.863573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.863629 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.863633 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:36.863661 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:52:36.863694 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":32,"thread_start_us":63,"threads_started":1}
I20260927 02:52:36.863997 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.859463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35413 (local address 127.0.0.1:45698)
0927 02:52:36.859753 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:36.859763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:36.859824 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:36.860057 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:36.860060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:36.860062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:36.860119 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.860121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.861146 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.861147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.862438 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:36.862440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.863399 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.863400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:36.863460 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.863464 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:36.863510 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:52:36.863545 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":41,"threads_started":1}
I20260927 02:52:36.864008 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.859589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.0.98.2:45213)
0927 02:52:36.859935 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:36.859938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:36.859946 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:36.859979 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:36.859981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:36.859993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:36.860063 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:36.860269 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.861396 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.862870 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.863487 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.863756 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.863873 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:36.863898 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 02:52:36.863933 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":83,"threads_started":1}
I20260927 02:52:36.864265 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:36.859584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.0.98.1:35413)
0927 02:52:36.859781 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:36.859786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:36.859811 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:52:36.859884 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:36.859887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:36.859902 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:36.859972 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:36.860177 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.861064 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.862517 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:36.863261 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:36.863327 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:36.863367 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:36.864146 (+ 779us) server_negotiation.cc:299] Negotiation successful
0927 02:52:36.864191 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260927 02:52:36.864224 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec2b318147c4369a8552d2b8a5a2661 (DEFAULT_TABLE table=test-table [id=f88c7e438aa747efbcd6edce2a90a28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:36.864557 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec2b318147c4369a8552d2b8a5a2661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:36.864938 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec2b318147c4369a8552d2b8a5a2661 (DEFAULT_TABLE table=test-table [id=f88c7e438aa747efbcd6edce2a90a28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:36.865236 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec2b318147c4369a8552d2b8a5a2661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:36.867652 1435 tablet_bootstrap.cc:492] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287: Bootstrap starting.
I20260927 02:52:36.868287 1436 tablet_bootstrap.cc:492] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d: Bootstrap starting.
I20260927 02:52:36.868312 1435 tablet_bootstrap.cc:654] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:36.868357 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec2b318147c4369a8552d2b8a5a2661 (DEFAULT_TABLE table=test-table [id=f88c7e438aa747efbcd6edce2a90a28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:36.868748 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec2b318147c4369a8552d2b8a5a2661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:36.869228 1436 tablet_bootstrap.cc:654] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:36.869334 1435 log.cc:824] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:36.869743 1436 log.cc:824] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:36.870241 1435 tablet_bootstrap.cc:492] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287: No bootstrap required, opened a new log
I20260927 02:52:36.870312 1435 ts_tablet_manager.cc:1397] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:52:36.870453 1436 tablet_bootstrap.cc:492] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d: No bootstrap required, opened a new log
I20260927 02:52:36.870508 1436 ts_tablet_manager.cc:1397] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:52:36.871856 1439 tablet_bootstrap.cc:492] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d: Bootstrap starting.
I20260927 02:52:36.871893 1436 raft_consensus.cc:348] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:36.872058 1436 raft_consensus.cc:374] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:36.872094 1436 raft_consensus.cc:729] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5719f6d657417b92b26c2b3c9a3f0d, State: Initialized, Role: FOLLOWER
I20260927 02:52:36.872134 1435 raft_consensus.cc:348] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:36.872244 1436 consensus_queue.cc:260] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:36.872288 1435 raft_consensus.cc:374] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:36.872327 1435 raft_consensus.cc:729] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c81b1914ed947dc895b5dff8704a287, State: Initialized, Role: FOLLOWER
I20260927 02:52:36.872452 1435 consensus_queue.cc:260] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:36.872478 1436 ts_tablet_manager.cc:1428] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:52:36.872545 1439 tablet_bootstrap.cc:654] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:36.872586 1153 heartbeater.cc:503] Master 127.0.98.62:33541 was elected leader, sending a full tablet report...
I20260927 02:52:36.872696 1435 ts_tablet_manager.cc:1428] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 02:52:36.872726 1421 heartbeater.cc:503] Master 127.0.98.62:33541 was elected leader, sending a full tablet report...
I20260927 02:52:36.873075 1439 log.cc:824] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:36.873720 1439 tablet_bootstrap.cc:492] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d: No bootstrap required, opened a new log
I20260927 02:52:36.873780 1439 ts_tablet_manager.cc:1397] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:52:36.875458 1439 raft_consensus.cc:348] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:36.875578 1439 raft_consensus.cc:374] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:36.875602 1439 raft_consensus.cc:729] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482bceab887f4444bc4f8c356945890d, State: Initialized, Role: FOLLOWER
I20260927 02:52:36.875680 1439 consensus_queue.cc:260] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:36.875886 1439 ts_tablet_manager.cc:1428] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:52:36.875919 1287 heartbeater.cc:503] Master 127.0.98.62:33541 was elected leader, sending a full tablet report...
W20260927 02:52:36.932194 1288 tablet.cc:2369] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:52:37.017095 1441 raft_consensus.cc:482] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:37.017220 1441 raft_consensus.cc:504] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:37.017634 1441 leader_election.cc:288] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 482bceab887f4444bc4f8c356945890d (127.0.98.2:45213), cb5719f6d657417b92b26c2b3c9a3f0d (127.0.98.1:35413)
I20260927 02:52:37.020280 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:37.017883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36551 (local address 127.0.98.2:45213)
0927 02:52:37.017931 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:37.017933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:37.017943 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:52:37.018029 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:37.018031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:37.018042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:37.018088 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:37.018223 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.018678 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.019601 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.019973 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.020029 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:37.020040 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:37.020180 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:52:37.020216 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:52:37.020280 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:37.017824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45213 (local address 127.0.98.3:36551)
0927 02:52:37.017879 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:37.017890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:37.017937 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:37.018099 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:37.018100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:37.018101 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:37.018161 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:37.018163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.018746 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.018747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:37.019517 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:37.019519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.020041 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.020042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:37.020117 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:37.020120 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:37.020168 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:52:37.020204 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":6}
I20260927 02:52:37.020349 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:37.017950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42447 (local address 127.0.98.1:35413)
0927 02:52:37.017987 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:37.017989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:37.018088 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:52:37.018157 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:37.018158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:37.018169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:37.018209 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:37.018341 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.018827 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.019680 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.020043 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.020100 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:37.020109 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:37.020243 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:52:37.020279 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:52:37.020406 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:37.017859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35413 (local address 127.0.98.3:42447)
0927 02:52:37.018023 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:37.018033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:37.018096 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:37.018220 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:37.018221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:37.018222 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:37.018283 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:37.018285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.018897 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.018898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:37.019607 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:37.019608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.020115 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.020116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:37.020185 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:37.020188 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:37.020231 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:52:37.020265 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260927 02:52:37.020675 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec2b318147c4369a8552d2b8a5a2661" candidate_uuid: "9c81b1914ed947dc895b5dff8704a287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" is_pre_election: true
I20260927 02:52:37.020675 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec2b318147c4369a8552d2b8a5a2661" candidate_uuid: "9c81b1914ed947dc895b5dff8704a287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482bceab887f4444bc4f8c356945890d" is_pre_election: true
I20260927 02:52:37.020831 1241 raft_consensus.cc:2454] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c81b1914ed947dc895b5dff8704a287 in term 0.
I20260927 02:52:37.020831 1107 raft_consensus.cc:2454] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c81b1914ed947dc895b5dff8704a287 in term 0.
I20260927 02:52:37.021035 1310 leader_election.cc:302] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c81b1914ed947dc895b5dff8704a287, cb5719f6d657417b92b26c2b3c9a3f0d; no voters:
I20260927 02:52:37.021155 1441 raft_consensus.cc:2781] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:37.021201 1441 raft_consensus.cc:482] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:37.021220 1441 raft_consensus.cc:3037] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:37.021883 1441 raft_consensus.cc:504] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:37.021992 1441 leader_election.cc:288] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [CANDIDATE]: Term 1 election: Requested vote from peers 482bceab887f4444bc4f8c356945890d (127.0.98.2:45213), cb5719f6d657417b92b26c2b3c9a3f0d (127.0.98.1:35413)
I20260927 02:52:37.022167 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec2b318147c4369a8552d2b8a5a2661" candidate_uuid: "9c81b1914ed947dc895b5dff8704a287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d"
I20260927 02:52:37.022248 1107 raft_consensus.cc:3037] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:37.022223 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec2b318147c4369a8552d2b8a5a2661" candidate_uuid: "9c81b1914ed947dc895b5dff8704a287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482bceab887f4444bc4f8c356945890d"
I20260927 02:52:37.022310 1241 raft_consensus.cc:3037] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:37.022990 1107 raft_consensus.cc:2454] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c81b1914ed947dc895b5dff8704a287 in term 1.
I20260927 02:52:37.023084 1241 raft_consensus.cc:2454] T 4ec2b318147c4369a8552d2b8a5a2661 P 482bceab887f4444bc4f8c356945890d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c81b1914ed947dc895b5dff8704a287 in term 1.
I20260927 02:52:37.023150 1310 leader_election.cc:302] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c81b1914ed947dc895b5dff8704a287, cb5719f6d657417b92b26c2b3c9a3f0d; no voters:
I20260927 02:52:37.023221 1441 raft_consensus.cc:2781] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:37.023343 1441 raft_consensus.cc:686] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [term 1 LEADER]: Becoming Leader. State: Replica: 9c81b1914ed947dc895b5dff8704a287, State: Running, Role: LEADER
I20260927 02:52:37.023422 1441 consensus_queue.cc:237] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } }
I20260927 02:52:37.024286 962 catalog_manager.cc:5495] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c81b1914ed947dc895b5dff8704a287 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c81b1914ed947dc895b5dff8704a287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482bceab887f4444bc4f8c356945890d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb5719f6d657417b92b26c2b3c9a3f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c81b1914ed947dc895b5dff8704a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35589 } health_report { overall_health: HEALTHY } } }
W20260927 02:52:37.040172 1154 tablet.cc:2369] T 4ec2b318147c4369a8552d2b8a5a2661 P cb5719f6d657417b92b26c2b3c9a3f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:52:37 dist-test-slave-mfx9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:52:37.049146 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:37.042008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.0.98.1:35413)
0927 02:52:37.042066 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:37.042068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:37.042162 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:52:37.042230 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:37.042231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:37.042245 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:37.042290 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:37.042838 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.043308 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.043988 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.044171 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.044226 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:37.046562 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:37.046573 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:37.046573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:37.046580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:37.048156 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:37.048453 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:37.048457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:37.048457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:37.048526 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:37.048676 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:37.048676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:37.048676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:37.048843 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:37.048949 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:37.049018 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:52:37.049078 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 02:52:37 dist-test-slave-mfx9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:52:37 dist-test-slave-mfx9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:52:37.070350 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:37.064117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.0.98.1:35413)
0927 02:52:37.064177 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:37.064179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:37.064243 (+ 64us) server_negotiation.cc:407] Connection header received
0927 02:52:37.064311 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:37.064312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:37.064323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:37.064368 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:37.064855 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.065316 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.066010 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.066191 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.066243 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:37.068287 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:37.068288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:37.068288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:37.068295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:37.069366 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:37.069641 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:37.069642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:37.069642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:37.069689 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:37.069857 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:37.069858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:37.069858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:37.069945 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:37.070025 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:37.070185 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 02:52:37.070260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 02:52:37 dist-test-slave-mfx9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 02:52:37.083320 1422 tablet.cc:2369] T 4ec2b318147c4369a8552d2b8a5a2661 P 9c81b1914ed947dc895b5dff8704a287: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:52:37 dist-test-slave-mfx9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:52:37.090267 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:37.083928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.0.98.1:35413)
0927 02:52:37.083991 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:37.083994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:37.084075 (+ 81us) server_negotiation.cc:407] Connection header received
0927 02:52:37.084141 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:37.084142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:37.084152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:37.084196 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:37.084710 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.085187 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.085885 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:37.086068 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:37.086122 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:37.088155 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:37.088157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:37.088157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:37.088166 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:37.089331 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:37.089617 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:37.089618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:37.089619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:37.089668 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:37.089824 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:37.089825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:37.089825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:37.089917 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:37.090000 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:37.090100 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:52:37.090175 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 02:52:37.090687 1067 tablet_service.cc:2908] Rejecting scan request for tablet 4ec2b318147c4369a8552d2b8a5a2661: Uninitialized: clean time has not yet been initialized
I20260927 02:52:37.091454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 1024
I20260927 02:52:37.095919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 1158
I20260927 02:52:37.100534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 1292
I20260927 02:52:37.104554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 930
2026-09-27T02:52:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1942 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:38 dist-test-slave-mfx9 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:38 dist-test-slave-mfx9 krb5kdc[1459](info): set up 2 sockets
Sep 27 02:52:38 dist-test-slave-mfx9 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260927 02:52:39.531109 392 mini_kdc.cc:121] Time spent starting KDC: real 2.406s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 02:52:39.674290 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.143s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 02:52:39.777453 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.103s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 02:52:39.886526 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.109s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/test-user.keytab.
W20260927 02:52:40.428153 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.542s user 0.000s sys 0.001s
Sep 27 02:52:40 dist-test-slave-mfx9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260927 02:52:41.057137 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.610s user 0.000s sys 0.003s
I20260927 02:52:41.057528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:41.138629 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:41.138888 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:41.138916 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:41.140192 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:41.140237 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:52:41.140250 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:41.140261 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:41.140273 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:41.140295 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:41.141925 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46767
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:41.142130 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:41.142302 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:41.146912 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:41.146958 1472 server_base.cc:1034] running on GCE node
W20260927 02:52:41.146925 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:41.147012 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:41.147377 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:41.147755 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:41.148921 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790477561148903 us; error 34 us; skew 500 ppm
Sep 27 02:52:41 dist-test-slave-mfx9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:41.151782 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:41.152228 1472 webserver.cc:460] Webserver started at http://127.0.98.62:43193/ using document root <none> and password file <none>
I20260927 02:52:41.152407 1472 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:41.152446 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:41.152541 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:41.153515 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "6c0055fc967f4f0690d431dc7b611ef3"
format_stamp: "Formatted at 2026-09-27 02:52:41 on dist-test-slave-mfx9"
server_key: "bcd7559796036793e81eb7a1b3fcd29e"
server_key_iv: "0719d4e139153814a526ad19dfd7a8e9"
server_key_version: "encryptionkey@0"
I20260927 02:52:41.153790 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c0055fc967f4f0690d431dc7b611ef3"
format_stamp: "Formatted at 2026-09-27 02:52:41 on dist-test-slave-mfx9"
server_key: "bcd7559796036793e81eb7a1b3fcd29e"
server_key_iv: "0719d4e139153814a526ad19dfd7a8e9"
server_key_version: "encryptionkey@0"
I20260927 02:52:41.236861 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.083s user 0.001s sys 0.002s
I20260927 02:52:41.237633 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:41.237802 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:41.237898 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "6c0055fc967f4f0690d431dc7b611ef3"
format_stamp: "Formatted at 2026-09-27 02:52:41 on dist-test-slave-mfx9"
server_key: "bcd7559796036793e81eb7a1b3fcd29e"
server_key_iv: "0719d4e139153814a526ad19dfd7a8e9"
server_key_version: "encryptionkey@0"
I20260927 02:52:41.237967 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:41.252010 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:41.253744 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:41.253850 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:41.258111 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46767
I20260927 02:52:41.258158 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46767 every 8 connection(s)
I20260927 02:52:41.258478 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:41.259150 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:41.261808 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3: Bootstrap starting.
I20260927 02:52:41.262398 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:41.262655 1550 log.cc:824] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:41.263203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 1472
I20260927 02:52:41.263291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:41.263406 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3: No bootstrap required, opened a new log
I20260927 02:52:41.263449 392 external_mini_cluster.cc:1428] Setting key 96fd7fbdbc294db9c2349d8b99d6f8b4
I20260927 02:52:41.265957 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0055fc967f4f0690d431dc7b611ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } }
I20260927 02:52:41.266135 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:41.266168 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0055fc967f4f0690d431dc7b611ef3, State: Initialized, Role: FOLLOWER
I20260927 02:52:41.266326 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0055fc967f4f0690d431dc7b611ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } }
I20260927 02:52:41.266405 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:41.266459 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:41.266502 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:41.267210 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0055fc967f4f0690d431dc7b611ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } }
I20260927 02:52:41.267379 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c0055fc967f4f0690d431dc7b611ef3; no voters:
I20260927 02:52:41.267536 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:41.267572 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:41.267755 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 6c0055fc967f4f0690d431dc7b611ef3, State: Running, Role: LEADER
I20260927 02:52:41.267866 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0055fc967f4f0690d431dc7b611ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } }
I20260927 02:52:41.267931 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:41.268244 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c0055fc967f4f0690d431dc7b611ef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0055fc967f4f0690d431dc7b611ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } }
I20260927 02:52:41.268316 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:41.268455 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0055fc967f4f0690d431dc7b611ef3. Latest consensus state: current_term: 1 leader_uuid: "6c0055fc967f4f0690d431dc7b611ef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0055fc967f4f0690d431dc7b611ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } }
I20260927 02:52:41.268535 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3 [sys.catalog]: This master's current role is: LEADER
W20260927 02:52:41.269882 1571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:52:41.269924 1571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:52:41.269963 1559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:41.270098 1559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:52:41 dist-test-slave-mfx9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:41.272703 1559 catalog_manager.cc:1347] Generated new cluster ID: abb1df815f3649928e75f5b223ba9d29
I20260927 02:52:41.272754 1559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:41.274597 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:41.263940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.0.98.62:46767)
0927 02:52:41.264098 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:41.264107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:41.264134 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:52:41.264639 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:41.264649 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:41.264713 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:41.264882 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:41.265560 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:41.266893 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:41.267880 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:41.268444 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:41.268523 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:41.271529 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:41.271533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:41.271533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:41.271544 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:41.273247 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:41.273548 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:41.273551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:41.273552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:41.273635 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:41.273823 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:41.273824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:41.273824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:41.274047 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:41.274185 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:41.274290 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:52:41.274416 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260927 02:52:41.296753 1559 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:41.297303 1559 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:41.307081 1559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3: Generated new TSK 0
I20260927 02:52:41.307261 1559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260927 02:52:42.260720 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.949s user 0.001s sys 0.002s
I20260927 02:52:42.261039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46767
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:42.342043 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:42.342248 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:42.342271 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:42.343393 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:42.343431 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:42.343451 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:52:42.343466 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:42.344913 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:42.345088 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:42.345259 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:42.350373 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:42.350373 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:42.350386 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:42.350487 1576 server_base.cc:1034] running on GCE node
I20260927 02:52:42.350816 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:42.351022 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:42.352205 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790477562352190 us; error 29 us; skew 500 ppm
Sep 27 02:52:42 dist-test-slave-mfx9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:42.354570 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:52:42.355127 1576 webserver.cc:460] Webserver started at http://127.0.98.1:35155/ using document root <none> and password file <none>
I20260927 02:52:42.355317 1576 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:42.355367 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:42.355484 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:42.356526 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b5a363a714742c2a057b33f2ab6e3b0"
format_stamp: "Formatted at 2026-09-27 02:52:42 on dist-test-slave-mfx9"
server_key: "4f023b8f57bd5e6f45bc1cad59324c52"
server_key_iv: "734b530831b41a5e4fe25dda787427a1"
server_key_version: "encryptionkey@0"
I20260927 02:52:42.356843 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b5a363a714742c2a057b33f2ab6e3b0"
format_stamp: "Formatted at 2026-09-27 02:52:42 on dist-test-slave-mfx9"
server_key: "4f023b8f57bd5e6f45bc1cad59324c52"
server_key_iv: "734b530831b41a5e4fe25dda787427a1"
server_key_version: "encryptionkey@0"
I20260927 02:52:42.358024 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:52:42.358716 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:42.358915 1576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:42.359002 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "1b5a363a714742c2a057b33f2ab6e3b0"
format_stamp: "Formatted at 2026-09-27 02:52:42 on dist-test-slave-mfx9"
server_key: "4f023b8f57bd5e6f45bc1cad59324c52"
server_key_iv: "734b530831b41a5e4fe25dda787427a1"
server_key_version: "encryptionkey@0"
I20260927 02:52:42.359064 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:42.370558 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:42.372485 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:42.372629 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:42.373126 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:42.373164 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:42.373193 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:42.373209 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:52:42 dist-test-slave-mfx9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:42.380215 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39535
I20260927 02:52:42.380551 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:52:42.380592 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39535 every 8 connection(s)
I20260927 02:52:42.382426 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:42.373466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33329 (local address 127.0.98.62:46767)
0927 02:52:42.373584 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:42.373586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:42.373749 (+ 163us) server_negotiation.cc:407] Connection header received
0927 02:52:42.374391 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:42.374392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:42.374402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:42.374450 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:42.375276 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.375852 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.377309 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.377538 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.377601 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:42.379807 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:42.379809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:42.379809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:42.379816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:42.381240 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:42.381585 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:42.381586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:42.381586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:42.381636 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:42.381843 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:42.381844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:42.381844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:42.381948 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:42.382003 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:42.382240 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 02:52:42.382331 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260927 02:52:42.382553 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:42.373451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.98.1:33329)
0927 02:52:42.373613 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:42.373660 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:42.374299 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:42.374539 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:42.374543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:42.374893 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:42.375177 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:42.375184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.375921 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.375923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:42.377206 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:42.377220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.377635 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.377636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:42.377749 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:42.378237 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:42.378247 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:42.379721 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:42.381311 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:42.381317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:42.381318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:42.381511 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:42.381689 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:42.381690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:42.381690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:42.381788 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:42.382001 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:42.382007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:42.382129 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 02:52:42.382212 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:52:42.382355 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260927 02:52:42.383028 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46767
I20260927 02:52:42.383106 1708 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:42.383278 1708 heartbeater.cc:511] Master 127.0.98.62:46767 requested a full tablet report, sending...
I20260927 02:52:42.383766 1503 ts_manager.cc:194] Registered new tserver with Master: 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535)
I20260927 02:52:42.384634 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33329
I20260927 02:52:42.385771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 1576
I20260927 02:52:42.385869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:42.386029 392 external_mini_cluster.cc:1428] Setting key 652811a57d9774456f96368773186678
I20260927 02:52:42.388962 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:42.383997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38641 (local address 127.0.98.62:46767)
0927 02:52:42.384034 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:42.384036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:42.384122 (+ 86us) server_negotiation.cc:407] Connection header received
0927 02:52:42.384184 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:42.384185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:42.384192 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:42.384236 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:42.384532 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.385107 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.385828 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.386059 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.386126 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:42.386940 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:42.386942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:42.386942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:42.386948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:42.388041 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:42.388324 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:42.388325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:42.388326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:42.388380 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:42.388530 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:42.388531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:42.388531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:42.388627 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:42.388678 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:42.388813 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:52:42.388878 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:52:42.388978 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:42.383947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.98.1:38641)
0927 02:52:42.384068 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:42.384078 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:42.384135 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:42.384241 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:42.384241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:42.384409 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:42.384473 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:42.384474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.385191 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.385192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:42.385750 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:42.385753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.386127 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.386128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:42.386188 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:42.386515 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:42.386519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:42.386865 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:42.388113 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:42.388114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:42.388114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:42.388266 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:42.388429 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:42.388430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:42.388430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:42.388482 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:42.388676 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:42.388677 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:42.388728 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 02:52:42.388786 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:52:42.388876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 02:52:42.873749 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.487s user 0.000s sys 0.003s
I20260927 02:52:42.874069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46767
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:42.951164 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:42.951351 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:42.951372 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:42.952513 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:42.952556 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:42.952576 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:52:42.952592 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:42.954082 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:42.954293 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:42.954505 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:42.959275 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:42.959287 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:42.959457 1717 server_base.cc:1034] running on GCE node
W20260927 02:52:42.959347 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:42.959815 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:42.960017 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:42.961180 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790477562961164 us; error 28 us; skew 500 ppm
Sep 27 02:52:42 dist-test-slave-mfx9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:42.963639 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:52:42.964084 1717 webserver.cc:460] Webserver started at http://127.0.98.2:40191/ using document root <none> and password file <none>
I20260927 02:52:42.964282 1717 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:42.964342 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:42.964465 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:42.965461 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "93da5f502f06428b865c05b361f78435"
format_stamp: "Formatted at 2026-09-27 02:52:42 on dist-test-slave-mfx9"
server_key: "be33c00027dbae6e580ce438c1e42bcf"
server_key_iv: "0882b11f9040ed6b651176aad0e10e16"
server_key_version: "encryptionkey@0"
I20260927 02:52:42.965767 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "93da5f502f06428b865c05b361f78435"
format_stamp: "Formatted at 2026-09-27 02:52:42 on dist-test-slave-mfx9"
server_key: "be33c00027dbae6e580ce438c1e42bcf"
server_key_iv: "0882b11f9040ed6b651176aad0e10e16"
server_key_version: "encryptionkey@0"
I20260927 02:52:42.966966 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:42.967623 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:42.967792 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:42.967873 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "93da5f502f06428b865c05b361f78435"
format_stamp: "Formatted at 2026-09-27 02:52:42 on dist-test-slave-mfx9"
server_key: "be33c00027dbae6e580ce438c1e42bcf"
server_key_iv: "0882b11f9040ed6b651176aad0e10e16"
server_key_version: "encryptionkey@0"
I20260927 02:52:42.967922 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:42.977643 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:42.979373 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:42.979473 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:42.979880 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:42.979916 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:42.979934 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:42.979946 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:42.986181 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41769
I20260927 02:52:42.986267 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41769 every 8 connection(s)
I20260927 02:52:42.986547 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:52:42 dist-test-slave-mfx9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:42.988076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 1717
I20260927 02:52:42.988159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:42.988350 392 external_mini_cluster.cc:1428] Setting key 9419ea2a0df184447226ce12ebce01e5
I20260927 02:52:42.989600 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:42.980333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41677 (local address 127.0.98.62:46767)
0927 02:52:42.980453 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:42.980455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:42.980626 (+ 171us) server_negotiation.cc:407] Connection header received
0927 02:52:42.981187 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:42.981188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:42.981203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:42.981271 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:42.982113 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.982937 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.984439 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.984667 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.984720 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:42.987145 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:42.987146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:42.987147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:42.987153 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:42.988235 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:42.988629 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:42.988630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:42.988631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:42.988696 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:42.988961 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:42.988963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:42.988963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:42.989090 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:42.989160 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:42.989418 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:52:42.989504 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260927 02:52:42.989727 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:42.980279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.98.2:41677)
0927 02:52:42.980511 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:42.980548 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:42.981058 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:42.981381 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:42.981386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:42.981724 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:42.982000 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:42.982007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.983014 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.983015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:42.984350 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:42.984353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.984745 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.984746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:42.984893 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:42.985431 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:42.985441 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:42.987064 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:42.988309 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:42.988315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:42.988317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:42.988578 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:42.988749 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:42.988749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:42.988750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:42.988899 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:42.989147 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:42.989152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:42.989293 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 02:52:42.989382 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:52:42.989526 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260927 02:52:42.990152 1849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46767
I20260927 02:52:42.990235 1849 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:42.990402 1849 heartbeater.cc:511] Master 127.0.98.62:46767 requested a full tablet report, sending...
I20260927 02:52:42.990710 1504 ts_manager.cc:194] Registered new tserver with Master: 93da5f502f06428b865c05b361f78435 (127.0.98.2:41769)
I20260927 02:52:42.991195 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41677
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:52:42.996155 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:42.991259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53423 (local address 127.0.98.62:46767)
0927 02:52:42.991335 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:42.991337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:42.991344 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:52:42.991438 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:42.991439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:42.991450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:42.991501 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:42.991778 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.992292 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.992939 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.993140 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.993190 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:42.993949 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:42.993951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:42.993951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:42.993957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:42.995211 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:42.995482 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:42.995483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:42.995484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:42.995533 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:42.995682 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:42.995682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:42.995683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:42.995815 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:42.995871 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:42.996003 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:52:42.996069 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:52:42.996168 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:42.991115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.98.2:53423)
0927 02:52:42.991250 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:42.991261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:42.991367 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:42.991505 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:42.991506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:42.991661 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:42.991715 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:42.991716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.992372 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.992373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:42.992873 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:42.992874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:42.993195 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:42.993196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:42.993257 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:42.993619 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:42.993624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:42.993875 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:42.995277 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:42.995278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:42.995279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:42.995427 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:42.995583 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:42.995583 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:42.995584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:42.995635 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:42.995876 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:42.995877 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:42.995917 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:52:42.995976 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:52:42.996068 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:52:43.011081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46767
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:43.089247 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:43.089460 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:43.089493 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:43.090692 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:43.090734 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:43.090780 1858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:52:43.090804 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:43.092329 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:43.092541 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:43.092752 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:43.097514 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:43.097591 1858 server_base.cc:1034] running on GCE node
W20260927 02:52:43.097545 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:43.097654 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:43.097875 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:43.098095 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:43.099227 1858 hybrid_clock.cc:648] HybridClock initialized: now 1790477563099207 us; error 32 us; skew 500 ppm
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:43.101480 1858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:52:43.101939 1858 webserver.cc:460] Webserver started at http://127.0.98.3:41467/ using document root <none> and password file <none>
I20260927 02:52:43.102137 1858 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:43.102183 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:43.102301 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:43.103343 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "b194bddd4b734fe2a2f2896984cce6bc"
format_stamp: "Formatted at 2026-09-27 02:52:43 on dist-test-slave-mfx9"
server_key: "fdcde3098023318b3cb14e246aefcd30"
server_key_iv: "bf9e67634090845483cc6bed71c3f50e"
server_key_version: "encryptionkey@0"
I20260927 02:52:43.103664 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b194bddd4b734fe2a2f2896984cce6bc"
format_stamp: "Formatted at 2026-09-27 02:52:43 on dist-test-slave-mfx9"
server_key: "fdcde3098023318b3cb14e246aefcd30"
server_key_iv: "bf9e67634090845483cc6bed71c3f50e"
server_key_version: "encryptionkey@0"
I20260927 02:52:43.104820 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:52:43.105508 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:43.105646 1858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:43.105722 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "b194bddd4b734fe2a2f2896984cce6bc"
format_stamp: "Formatted at 2026-09-27 02:52:43 on dist-test-slave-mfx9"
server_key: "fdcde3098023318b3cb14e246aefcd30"
server_key_iv: "bf9e67634090845483cc6bed71c3f50e"
server_key_version: "encryptionkey@0"
I20260927 02:52:43.105777 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:43.126323 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:43.128131 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:43.128269 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:43.128770 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:43.128809 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:43.128839 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:43.128865 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:43.135097 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36705
I20260927 02:52:43.135249 1988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36705 every 8 connection(s)
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:43.135506 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:52:43.138116 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.129222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37037 (local address 127.0.98.62:46767)
0927 02:52:43.129282 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.129285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.129475 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:52:43.129962 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.129964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.129976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.130051 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:43.130862 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.131595 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.133026 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.133323 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.133398 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.135834 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:43.135835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:43.135835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:43.135842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:43.136933 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.137289 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.137290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.137290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.137334 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.137490 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.137491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.137491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.137581 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:43.137624 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.137991 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.138055 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:52:43.138286 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.129193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.98.3:37037)
0927 02:52:43.129350 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.129391 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.129872 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.130123 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.130126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.130469 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:43.130741 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.130748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.131698 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.131698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.132951 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.132954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.133399 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.133400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.133494 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.134032 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:43.134042 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:43.135719 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:43.136996 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:43.137001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:43.137002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:43.137232 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:43.137370 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:43.137370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:43.137370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:43.137452 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:43.137750 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:43.137756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:43.137878 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.137973 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.138092 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260927 02:52:43.138711 1989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46767
I20260927 02:52:43.138800 1989 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:43.138964 1989 heartbeater.cc:511] Master 127.0.98.62:46767 requested a full tablet report, sending...
I20260927 02:52:43.139335 1503 ts_manager.cc:194] Registered new tserver with Master: b194bddd4b734fe2a2f2896984cce6bc (127.0.98.3:36705)
I20260927 02:52:43.139703 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37037
I20260927 02:52:43.144385 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.139673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34327 (local address 127.0.98.62:46767)
0927 02:52:43.139723 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.139725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.139769 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:52:43.139829 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.139829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.139839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.139885 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:43.140182 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.140764 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.141412 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.141589 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.141637 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.142299 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:43.142300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:43.142300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:43.142305 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:43.143573 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.143833 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.143834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.143834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.143874 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.143990 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.143990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.143991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.144071 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:43.144113 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.144282 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.144330 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:52:43.144413 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.139612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.98.3:34327)
0927 02:52:43.139720 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.139730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.139782 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.139889 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.139889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.140063 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:43.140115 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.140116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.140838 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.140839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.141350 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.141352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.141639 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.141639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.141680 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.141998 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:43.142002 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:43.142237 (+ 235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:43.143643 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:43.143644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:43.143644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:43.143785 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:43.143915 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:43.143915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:43.143915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:43.143958 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:43.144144 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:43.144145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:43.144186 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.144266 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.144345 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":64,"threads_started":1}
I20260927 02:52:43.145447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 1858
I20260927 02:52:43.145535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:43.145715 392 external_mini_cluster.cc:1428] Setting key d7e7c923aa091ba1169b640e40c5e71a
I20260927 02:52:43.146616 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:43.167907 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.160443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.0.98.62:46767)
0927 02:52:43.160493 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.160496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.160547 (+ 51us) server_negotiation.cc:407] Connection header received
0927 02:52:43.160617 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.160618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.160630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.160685 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:43.161296 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.162156 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.163224 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.163522 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.163584 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.165883 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:43.165884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:43.165885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:43.165892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:43.167033 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.167305 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.167305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.167306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.167346 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.167525 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.167526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.167526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.167637 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:43.167709 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.167778 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.167836 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:52:43.169291 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:52:43.170862 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:43.176971 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.173106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.0.98.1:39535)
0927 02:52:43.173212 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.173217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.173308 (+ 91us) server_negotiation.cc:407] Connection header received
0927 02:52:43.173406 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.173409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.173425 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.173502 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:43.173672 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.174871 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.176139 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.176612 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.176670 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.176725 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.176850 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.176896 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":58,"threads_started":1}
I20260927 02:52:43.176994 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.172988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.0.1:41488)
0927 02:52:43.173146 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.173187 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.173248 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.173597 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.173599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.173601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:43.173651 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.173653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.174931 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.174932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.175765 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.175768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.176717 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.176718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.176778 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.176782 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.176832 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.176899 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":29,"thread_start_us":69,"threads_started":1}
I20260927 02:52:43.176995 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.173010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39535 (local address 127.0.0.1:38332)
0927 02:52:43.173243 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.173252 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.173330 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.173516 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.173519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.173521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:43.173585 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.173587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.174937 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.174938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.176048 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.176050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.176683 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.176684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.176757 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.176770 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.176839 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.176903 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":49,"thread_start_us":88,"threads_started":1}
I20260927 02:52:43.177271 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.173531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.0.98.2:41769)
0927 02:52:43.173696 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.173700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.173710 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:52:43.173742 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.173744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.173758 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.173953 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:43.174006 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.174670 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.176100 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.176781 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.177033 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.177116 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.177143 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.177179 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260927 02:52:43.177273 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.173316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.0.98.3:36705)
0927 02:52:43.173455 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.173458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.173466 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:43.173500 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.173502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.173516 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.173585 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:43.173708 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.174841 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.175850 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.176550 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.176613 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.176684 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.177148 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.177183 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260927 02:52:43.177343 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.173186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41769 (local address 127.0.0.1:36376)
0927 02:52:43.173282 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.173289 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.173329 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.173856 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.173857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.173859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:43.173890 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.173891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.174746 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.174747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.176014 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.176017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.176880 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.176881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.176932 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.176936 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.176964 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.176996 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":63,"threads_started":1}
I20260927 02:52:43.177997 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 5e56b3df659543259732e9146f797a08 (DEFAULT_TABLE table=test-table [id=7a8373048c374f7cac6299746e23502c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:43.178361 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e56b3df659543259732e9146f797a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.178308 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 5e56b3df659543259732e9146f797a08 (DEFAULT_TABLE table=test-table [id=7a8373048c374f7cac6299746e23502c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:43.178414 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 5e56b3df659543259732e9146f797a08 (DEFAULT_TABLE table=test-table [id=7a8373048c374f7cac6299746e23502c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:43.178601 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e56b3df659543259732e9146f797a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.178714 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e56b3df659543259732e9146f797a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.180827 2013 tablet_bootstrap.cc:492] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: Bootstrap starting.
I20260927 02:52:43.181402 2013 tablet_bootstrap.cc:654] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.181665 2013 log.cc:824] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:43.181900 2014 tablet_bootstrap.cc:492] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: Bootstrap starting.
I20260927 02:52:43.182199 2016 tablet_bootstrap.cc:492] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Bootstrap starting.
I20260927 02:52:43.182500 2013 tablet_bootstrap.cc:492] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: No bootstrap required, opened a new log
I20260927 02:52:43.182523 2014 tablet_bootstrap.cc:654] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.182572 2013 ts_tablet_manager.cc:1397] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:52:43.182806 2016 tablet_bootstrap.cc:654] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.182850 2014 log.cc:824] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:43.183130 2016 log.cc:824] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:43.183619 2014 tablet_bootstrap.cc:492] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: No bootstrap required, opened a new log
I20260927 02:52:43.183678 2014 ts_tablet_manager.cc:1397] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:52:43.183779 2016 tablet_bootstrap.cc:492] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: No bootstrap required, opened a new log
I20260927 02:52:43.183825 2016 ts_tablet_manager.cc:1397] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:52:43.184988 2013 raft_consensus.cc:348] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.185138 2013 raft_consensus.cc:374] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.185079 2016 raft_consensus.cc:348] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.185169 2013 raft_consensus.cc:729] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5a363a714742c2a057b33f2ab6e3b0, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.185184 2016 raft_consensus.cc:374] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.185202 2016 raft_consensus.cc:729] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93da5f502f06428b865c05b361f78435, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.185281 2013 consensus_queue.cc:260] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.185307 2016 consensus_queue.cc:260] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.185499 2013 ts_tablet_manager.cc:1428] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:52:43.185534 1708 heartbeater.cc:503] Master 127.0.98.62:46767 was elected leader, sending a full tablet report...
I20260927 02:52:43.185489 2014 raft_consensus.cc:348] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.185614 2014 raft_consensus.cc:374] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.185644 2014 raft_consensus.cc:729] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b194bddd4b734fe2a2f2896984cce6bc, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.185735 2014 consensus_queue.cc:260] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.185894 1849 heartbeater.cc:503] Master 127.0.98.62:46767 was elected leader, sending a full tablet report...
I20260927 02:52:43.185961 2014 ts_tablet_manager.cc:1428] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:52:43.186056 1989 heartbeater.cc:503] Master 127.0.98.62:46767 was elected leader, sending a full tablet report...
I20260927 02:52:43.186270 2016 ts_tablet_manager.cc:1428] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260927 02:52:43.237125 1850 tablet.cc:2369] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:52:43.292732 2020 raft_consensus.cc:482] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:43.292860 2020 raft_consensus.cc:504] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.293211 2020 leader_election.cc:288] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535), b194bddd4b734fe2a2f2896984cce6bc (127.0.98.3:36705)
I20260927 02:52:43.296898 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.293526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49439 (local address 127.0.98.3:36705)
0927 02:52:43.293565 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.293566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.293653 (+ 87us) server_negotiation.cc:407] Connection header received
0927 02:52:43.293700 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.293701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.293710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.293748 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:43.293889 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.294602 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.295984 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.296556 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.296622 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.296632 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.296785 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.296824 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:52:43.296908 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.293434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49071 (local address 127.0.98.1:39535)
0927 02:52:43.293481 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.293483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.293493 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:52:43.293569 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.293571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.293582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.293623 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:43.293785 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.294525 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.295985 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.296546 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.296622 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.296632 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.296785 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.296826 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:52:43.296918 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.293406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.98.2:49439)
0927 02:52:43.293572 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.293582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.293647 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.293768 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.293769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.293770 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:43.293834 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.293835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.294667 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.294668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.295887 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.295890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.296626 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.296627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.296716 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.296720 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.296771 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.296816 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":13,"thread_start_us":108,"threads_started":1}
I20260927 02:52:43.296924 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.293386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39535 (local address 127.0.98.2:49071)
0927 02:52:43.293435 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.293447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.293485 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.293648 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.293649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.293650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:43.293720 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.293722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.294608 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.294609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.295887 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.295890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.296626 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.296627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.296716 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.296719 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.296771 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.296816 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":29}
I20260927 02:52:43.297290 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e56b3df659543259732e9146f797a08" candidate_uuid: "93da5f502f06428b865c05b361f78435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b194bddd4b734fe2a2f2896984cce6bc" is_pre_election: true
I20260927 02:52:43.297464 1943 raft_consensus.cc:2454] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93da5f502f06428b865c05b361f78435 in term 0.
I20260927 02:52:43.297451 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e56b3df659543259732e9146f797a08" candidate_uuid: "93da5f502f06428b865c05b361f78435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" is_pre_election: true
I20260927 02:52:43.297569 1662 raft_consensus.cc:2454] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93da5f502f06428b865c05b361f78435 in term 0.
I20260927 02:52:43.297694 1734 leader_election.cc:302] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93da5f502f06428b865c05b361f78435, b194bddd4b734fe2a2f2896984cce6bc; no voters:
I20260927 02:52:43.297814 2020 raft_consensus.cc:2781] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:43.297870 2020 raft_consensus.cc:482] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:43.297891 2020 raft_consensus.cc:3037] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.298792 2020 raft_consensus.cc:504] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.298918 2020 leader_election.cc:288] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [CANDIDATE]: Term 1 election: Requested vote from peers 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535), b194bddd4b734fe2a2f2896984cce6bc (127.0.98.3:36705)
I20260927 02:52:43.299067 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e56b3df659543259732e9146f797a08" candidate_uuid: "93da5f502f06428b865c05b361f78435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b194bddd4b734fe2a2f2896984cce6bc"
I20260927 02:52:43.299093 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e56b3df659543259732e9146f797a08" candidate_uuid: "93da5f502f06428b865c05b361f78435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b5a363a714742c2a057b33f2ab6e3b0"
I20260927 02:52:43.299149 1943 raft_consensus.cc:3037] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.299154 1662 raft_consensus.cc:3037] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.299816 1662 raft_consensus.cc:2454] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93da5f502f06428b865c05b361f78435 in term 1.
I20260927 02:52:43.299836 1943 raft_consensus.cc:2454] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93da5f502f06428b865c05b361f78435 in term 1.
I20260927 02:52:43.299979 1733 leader_election.cc:302] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5a363a714742c2a057b33f2ab6e3b0, 93da5f502f06428b865c05b361f78435; no voters:
I20260927 02:52:43.300107 2020 raft_consensus.cc:2781] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:43.300259 2020 raft_consensus.cc:686] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 1 LEADER]: Becoming Leader. State: Replica: 93da5f502f06428b865c05b361f78435, State: Running, Role: LEADER
I20260927 02:52:43.300367 2020 consensus_queue.cc:237] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } }
I20260927 02:52:43.301154 1503 catalog_manager.cc:5495] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93da5f502f06428b865c05b361f78435 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "93da5f502f06428b865c05b361f78435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } health_report { overall_health: HEALTHY } } }
I20260927 02:52:43.325989 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.323580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.98.2:41769)
0927 02:52:43.323629 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.323631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.323642 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:52:43.323740 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.323741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.323763 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.323825 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:43.323997 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.324464 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.325226 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.325417 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.325471 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.325668 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:43.325762 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:43.325828 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.325890 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.325910 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:52:43.327651 1662 raft_consensus.cc:1261] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Refusing update from remote peer 93da5f502f06428b865c05b361f78435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:43.328028 2020 consensus_queue.cc:1035] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:43.328140 1943 raft_consensus.cc:1261] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Refusing update from remote peer 93da5f502f06428b865c05b361f78435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:43.328382 2020 consensus_queue.cc:1035] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:43.330706 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333796099387551744 to 7333796099278839808. Current Snapshot: MvccSnapshot[applied={T|T < 7333796099387551744 or (T in {7333796099387551744})}]
I20260927 02:52:43.333369 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:43.333447 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:43.334097 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3: Sending DeleteTablet for 3 replicas of tablet 5e56b3df659543259732e9146f797a08
I20260927 02:52:43.334404 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e56b3df659543259732e9146f797a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41488
I20260927 02:52:43.334478 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e56b3df659543259732e9146f797a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36376
I20260927 02:52:43.334607 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e56b3df659543259732e9146f797a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38332
I20260927 02:52:43.334697 2036 tablet_replica.cc:330] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: stopping tablet replica
I20260927 02:52:43.334856 2036 raft_consensus.cc:2229] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:52:43.334877 2035 tablet_replica.cc:330] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: stopping tablet replica
I20260927 02:52:43.334947 2035 raft_consensus.cc:2229] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:43.334976 2036 raft_consensus.cc:2258] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:43.335067 2035 raft_consensus.cc:2258] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:43.335110 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:52:43.335237 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:43.335341 2036 ts_tablet_manager.cc:1905] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:43.335718 2035 ts_tablet_manager.cc:1905] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:43.336437 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 99a9f10dbb86466cb88a6c452c5fadff (DEFAULT_TABLE table=test-table [id=0f506d7f9c864e9eb687bf71825b3fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:43.336546 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a9f10dbb86466cb88a6c452c5fadff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.337002 2036 ts_tablet_manager.cc:1918] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:52:43.337059 2036 log.cc:1196] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/5e56b3df659543259732e9146f797a08
I20260927 02:52:43.337288 2036 ts_tablet_manager.cc:1939] T 5e56b3df659543259732e9146f797a08 P 93da5f502f06428b865c05b361f78435: Deleting consensus metadata
I20260927 02:52:43.337426 2037 tablet_replica.cc:330] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: stopping tablet replica
I20260927 02:52:43.337517 2037 raft_consensus.cc:2229] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:43.337603 2037 raft_consensus.cc:2258] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:43.337678 2035 ts_tablet_manager.cc:1918] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:52:43.337731 2035 log.cc:1196] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/5e56b3df659543259732e9146f797a08
I20260927 02:52:43.337812 2014 tablet_bootstrap.cc:492] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: Bootstrap starting.
I20260927 02:52:43.338184 2014 tablet_bootstrap.cc:654] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.338222 1488 catalog_manager.cc:4841] TS 93da5f502f06428b865c05b361f78435 (127.0.98.2:41769): tablet 5e56b3df659543259732e9146f797a08 (table test-table [id=7a8373048c374f7cac6299746e23502c]) successfully deleted
I20260927 02:52:43.338290 2037 ts_tablet_manager.cc:1905] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:43.338433 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 99a9f10dbb86466cb88a6c452c5fadff (DEFAULT_TABLE table=test-table [id=0f506d7f9c864e9eb687bf71825b3fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:43.338553 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a9f10dbb86466cb88a6c452c5fadff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.338634 2035 ts_tablet_manager.cc:1939] T 5e56b3df659543259732e9146f797a08 P b194bddd4b734fe2a2f2896984cce6bc: Deleting consensus metadata
I20260927 02:52:43.339171 2014 tablet_bootstrap.cc:492] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: No bootstrap required, opened a new log
I20260927 02:52:43.339210 2014 ts_tablet_manager.cc:1397] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:43.339349 2014 raft_consensus.cc:348] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.339408 2014 raft_consensus.cc:374] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.339424 2014 raft_consensus.cc:729] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b194bddd4b734fe2a2f2896984cce6bc, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.339450 2014 consensus_queue.cc:260] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.339526 2014 ts_tablet_manager.cc:1428] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:43.339826 2013 tablet_bootstrap.cc:492] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: Bootstrap starting.
I20260927 02:52:43.339964 1489 catalog_manager.cc:4841] TS b194bddd4b734fe2a2f2896984cce6bc (127.0.98.3:36705): tablet 5e56b3df659543259732e9146f797a08 (table test-table [id=7a8373048c374f7cac6299746e23502c]) successfully deleted
I20260927 02:52:43.340057 2037 ts_tablet_manager.cc:1918] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:52:43.340091 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 99a9f10dbb86466cb88a6c452c5fadff (DEFAULT_TABLE table=test-table [id=0f506d7f9c864e9eb687bf71825b3fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:43.340118 2037 log.cc:1196] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/5e56b3df659543259732e9146f797a08
I20260927 02:52:43.340183 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a9f10dbb86466cb88a6c452c5fadff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.340232 2013 tablet_bootstrap.cc:654] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.340333 2037 ts_tablet_manager.cc:1939] T 5e56b3df659543259732e9146f797a08 P 1b5a363a714742c2a057b33f2ab6e3b0: Deleting consensus metadata
I20260927 02:52:43.340937 2013 tablet_bootstrap.cc:492] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: No bootstrap required, opened a new log
I20260927 02:52:43.340967 2013 ts_tablet_manager.cc:1397] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:43.341061 2013 raft_consensus.cc:348] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.341100 2013 raft_consensus.cc:374] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.341111 2013 raft_consensus.cc:729] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5a363a714742c2a057b33f2ab6e3b0, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.341145 2013 consensus_queue.cc:260] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.341229 2013 ts_tablet_manager.cc:1428] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:43.341449 2016 tablet_bootstrap.cc:492] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: Bootstrap starting.
I20260927 02:52:43.341756 1488 catalog_manager.cc:4841] TS 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535): tablet 5e56b3df659543259732e9146f797a08 (table test-table [id=7a8373048c374f7cac6299746e23502c]) successfully deleted
I20260927 02:52:43.341805 2016 tablet_bootstrap.cc:654] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.342341 2016 tablet_bootstrap.cc:492] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: No bootstrap required, opened a new log
I20260927 02:52:43.342387 2016 ts_tablet_manager.cc:1397] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:43.342523 2016 raft_consensus.cc:348] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.342583 2016 raft_consensus.cc:374] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.342602 2016 raft_consensus.cc:729] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93da5f502f06428b865c05b361f78435, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.342643 2016 consensus_queue.cc:260] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.342718 2016 ts_tablet_manager.cc:1428] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260927 02:52:43.381446 1709 tablet.cc:2369] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:52:43.386010 1990 tablet.cc:2369] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:52:43.507532 2021 raft_consensus.cc:482] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:43.507644 2021 raft_consensus.cc:504] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.507958 2021 leader_election.cc:288] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535), 93da5f502f06428b865c05b361f78435 (127.0.98.2:41769)
I20260927 02:52:43.510854 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.508332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40541 (local address 127.0.98.2:41769)
0927 02:52:43.508381 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.508383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.508452 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:52:43.508532 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.508533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.508542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.508583 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:43.508729 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.509193 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.510175 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.510521 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.510578 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.510587 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.510724 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.510778 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:52:43.510867 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.508270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39053 (local address 127.0.98.1:39535)
0927 02:52:43.508347 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.508349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.508357 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:43.508386 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.508387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.508395 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.508445 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:43.508607 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.509118 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.510166 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.510540 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.510598 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.510609 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.510737 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.510793 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:52:43.510879 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.508181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39535 (local address 127.0.98.3:39053)
0927 02:52:43.508235 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.508249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.508320 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.508453 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.508454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.508456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:43.508546 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.508548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.509187 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.509188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.510086 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.510088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.510613 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.510614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.510678 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.510680 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.510732 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.510792 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:52:43.510982 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.508269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41769 (local address 127.0.98.3:40541)
0927 02:52:43.508389 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.508398 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.508453 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.508590 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.508591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.508592 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:43.508671 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.508672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.509264 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.509264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.510087 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.510089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.510600 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.510601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.510660 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.510664 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.510707 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.510746 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":7,"thread_start_us":66,"threads_started":1}
I20260927 02:52:43.511140 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a9f10dbb86466cb88a6c452c5fadff" candidate_uuid: "b194bddd4b734fe2a2f2896984cce6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" is_pre_election: true
I20260927 02:52:43.511149 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a9f10dbb86466cb88a6c452c5fadff" candidate_uuid: "b194bddd4b734fe2a2f2896984cce6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93da5f502f06428b865c05b361f78435" is_pre_election: true
I20260927 02:52:43.511229 1662 raft_consensus.cc:2454] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b194bddd4b734fe2a2f2896984cce6bc in term 0.
I20260927 02:52:43.511265 1803 raft_consensus.cc:2454] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b194bddd4b734fe2a2f2896984cce6bc in term 0.
I20260927 02:52:43.511404 1874 leader_election.cc:302] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5a363a714742c2a057b33f2ab6e3b0, b194bddd4b734fe2a2f2896984cce6bc; no voters:
I20260927 02:52:43.511515 2021 raft_consensus.cc:2781] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:43.511556 2021 raft_consensus.cc:482] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:43.511575 2021 raft_consensus.cc:3037] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.512204 2021 raft_consensus.cc:504] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.512312 2021 leader_election.cc:288] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [CANDIDATE]: Term 1 election: Requested vote from peers 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535), 93da5f502f06428b865c05b361f78435 (127.0.98.2:41769)
I20260927 02:52:43.512441 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a9f10dbb86466cb88a6c452c5fadff" candidate_uuid: "b194bddd4b734fe2a2f2896984cce6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b5a363a714742c2a057b33f2ab6e3b0"
I20260927 02:52:43.512507 1662 raft_consensus.cc:3037] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.512487 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a9f10dbb86466cb88a6c452c5fadff" candidate_uuid: "b194bddd4b734fe2a2f2896984cce6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93da5f502f06428b865c05b361f78435"
I20260927 02:52:43.512552 1803 raft_consensus.cc:3037] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.513077 1662 raft_consensus.cc:2454] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b194bddd4b734fe2a2f2896984cce6bc in term 1.
I20260927 02:52:43.513206 1803 raft_consensus.cc:2454] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b194bddd4b734fe2a2f2896984cce6bc in term 1.
I20260927 02:52:43.513234 1874 leader_election.cc:302] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5a363a714742c2a057b33f2ab6e3b0, b194bddd4b734fe2a2f2896984cce6bc; no voters:
I20260927 02:52:43.513370 2021 raft_consensus.cc:2781] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:43.513497 2021 raft_consensus.cc:686] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 1 LEADER]: Becoming Leader. State: Replica: b194bddd4b734fe2a2f2896984cce6bc, State: Running, Role: LEADER
I20260927 02:52:43.513553 2021 consensus_queue.cc:237] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.514250 1503 catalog_manager.cc:5495] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc reported cstate change: term changed from 0 to 1, leader changed from <none> to b194bddd4b734fe2a2f2896984cce6bc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } health_report { overall_health: HEALTHY } } }
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:43.561848 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.555411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.0.1:54906)
0927 02:52:43.555587 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.555595 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.555660 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.555772 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.555773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.556072 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:43.556122 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.556124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.556715 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.556716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.557232 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.557233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.557510 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.557511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.557545 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.557994 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:43.557999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:43.559593 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:43.560940 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:43.560946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:43.560949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:43.561137 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:43.561319 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:43.561319 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:43.561320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:43.561380 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:43.561575 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:43.561580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:43.561618 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.561667 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.561745 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260927 02:52:43.561905 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.555476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.0.98.62:46767)
0927 02:52:43.555532 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.555534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.555660 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:52:43.555701 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.555701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.555711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.555762 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:43.556189 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.556659 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.557292 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.557464 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.557508 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.559676 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:43.559677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:43.559677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:43.559683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:43.560870 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.561206 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.561207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.561208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.561261 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.561423 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.561424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.561424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.561532 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:43.561574 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.561687 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.561804 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:52:43.563627 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54906:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:52:43.563794 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:43.564994 1642 tablet_service.cc:1467] Processing CreateTablet for tablet d563245b03b940349ded32ed297b09b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=058ef5d1788d42ae9d35ba13b64a616a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:52:43.564997 1923 tablet_service.cc:1467] Processing CreateTablet for tablet d563245b03b940349ded32ed297b09b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=058ef5d1788d42ae9d35ba13b64a616a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:52:43.565045 1783 tablet_service.cc:1467] Processing CreateTablet for tablet d563245b03b940349ded32ed297b09b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=058ef5d1788d42ae9d35ba13b64a616a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:52:43.565150 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d563245b03b940349ded32ed297b09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.565148 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d563245b03b940349ded32ed297b09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.565150 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d563245b03b940349ded32ed297b09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:43.566167 2013 tablet_bootstrap.cc:492] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: Bootstrap starting.
I20260927 02:52:43.566522 2016 tablet_bootstrap.cc:492] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: Bootstrap starting.
I20260927 02:52:43.566877 2016 tablet_bootstrap.cc:654] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.567085 2013 tablet_bootstrap.cc:654] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.567399 2014 tablet_bootstrap.cc:492] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: Bootstrap starting.
I20260927 02:52:43.567409 2016 tablet_bootstrap.cc:492] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: No bootstrap required, opened a new log
I20260927 02:52:43.567454 2016 ts_tablet_manager.cc:1397] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:43.567571 2016 raft_consensus.cc:348] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.567623 2016 raft_consensus.cc:374] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.567634 2016 raft_consensus.cc:729] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93da5f502f06428b865c05b361f78435, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.567669 2016 consensus_queue.cc:260] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.567751 2016 ts_tablet_manager.cc:1428] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:43.567806 2014 tablet_bootstrap.cc:654] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:43.567803 2023 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.567888 2023 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:43.568456 2013 tablet_bootstrap.cc:492] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: No bootstrap required, opened a new log
I20260927 02:52:43.568465 2014 tablet_bootstrap.cc:492] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: No bootstrap required, opened a new log
I20260927 02:52:43.568502 2013 ts_tablet_manager.cc:1397] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:52:43.568501 2014 ts_tablet_manager.cc:1397] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:43.568630 2014 raft_consensus.cc:348] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.568629 2013 raft_consensus.cc:348] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.568686 2014 raft_consensus.cc:374] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.568703 2014 raft_consensus.cc:729] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b194bddd4b734fe2a2f2896984cce6bc, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.568701 2013 raft_consensus.cc:374] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:43.568720 2013 raft_consensus.cc:729] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5a363a714742c2a057b33f2ab6e3b0, State: Initialized, Role: FOLLOWER
I20260927 02:52:43.568748 2014 consensus_queue.cc:260] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.568760 2013 consensus_queue.cc:260] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.568830 2014 ts_tablet_manager.cc:1428] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:43.568856 2013 ts_tablet_manager.cc:1428] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:43.568883 2021 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.568962 2021 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:43.568975 2019 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.569042 2019 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:43.594996 2021 raft_consensus.cc:482] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:43.595082 2021 raft_consensus.cc:504] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.595212 2021 leader_election.cc:288] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535), 93da5f502f06428b865c05b361f78435 (127.0.98.2:41769)
I20260927 02:52:43.595440 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d563245b03b940349ded32ed297b09b0" candidate_uuid: "b194bddd4b734fe2a2f2896984cce6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93da5f502f06428b865c05b361f78435" is_pre_election: true
I20260927 02:52:43.595454 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d563245b03b940349ded32ed297b09b0" candidate_uuid: "b194bddd4b734fe2a2f2896984cce6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" is_pre_election: true
I20260927 02:52:43.595528 1803 raft_consensus.cc:2454] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b194bddd4b734fe2a2f2896984cce6bc in term 0.
I20260927 02:52:43.595556 1662 raft_consensus.cc:2454] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b194bddd4b734fe2a2f2896984cce6bc in term 0.
I20260927 02:52:43.595666 1874 leader_election.cc:302] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93da5f502f06428b865c05b361f78435, b194bddd4b734fe2a2f2896984cce6bc; no voters:
I20260927 02:52:43.595772 2021 raft_consensus.cc:2781] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:43.595817 2021 raft_consensus.cc:482] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:43.595836 2021 raft_consensus.cc:3037] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.596678 2021 raft_consensus.cc:504] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.596783 2021 leader_election.cc:288] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [CANDIDATE]: Term 1 election: Requested vote from peers 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535), 93da5f502f06428b865c05b361f78435 (127.0.98.2:41769)
I20260927 02:52:43.596923 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d563245b03b940349ded32ed297b09b0" candidate_uuid: "b194bddd4b734fe2a2f2896984cce6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b5a363a714742c2a057b33f2ab6e3b0"
I20260927 02:52:43.596984 1662 raft_consensus.cc:3037] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.596992 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d563245b03b940349ded32ed297b09b0" candidate_uuid: "b194bddd4b734fe2a2f2896984cce6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93da5f502f06428b865c05b361f78435"
I20260927 02:52:43.597074 1803 raft_consensus.cc:3037] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:43.597517 1662 raft_consensus.cc:2454] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b194bddd4b734fe2a2f2896984cce6bc in term 1.
I20260927 02:52:43.597673 1803 raft_consensus.cc:2454] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b194bddd4b734fe2a2f2896984cce6bc in term 1.
I20260927 02:52:43.597681 1874 leader_election.cc:302] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5a363a714742c2a057b33f2ab6e3b0, b194bddd4b734fe2a2f2896984cce6bc; no voters:
I20260927 02:52:43.597779 2021 raft_consensus.cc:2781] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:43.597838 2021 raft_consensus.cc:686] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [term 1 LEADER]: Becoming Leader. State: Replica: b194bddd4b734fe2a2f2896984cce6bc, State: Running, Role: LEADER
I20260927 02:52:43.597896 2021 consensus_queue.cc:237] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } }
I20260927 02:52:43.598193 2042 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: TxnStatusTablet state changed. Reason: New leader b194bddd4b734fe2a2f2896984cce6bc. Latest consensus state: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.598261 2042 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:52:43.598445 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:52:43.598552 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:52:43.598513 1504 catalog_manager.cc:5495] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc reported cstate change: term changed from 0 to 1, leader changed from <none> to b194bddd4b734fe2a2f2896984cce6bc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } health_report { overall_health: HEALTHY } } }
I20260927 02:52:43.605052 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.602833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.0.98.3:36705)
0927 02:52:43.602888 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.602890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.602924 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:52:43.603006 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.603007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.603024 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.603072 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:43.603193 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.603655 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.604363 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.604558 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.604610 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.604764 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:43.604854 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:43.604922 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.604968 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.604988 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:52:43.605073 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.602822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.0.1:41502)
0927 02:52:43.602879 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.602891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.602945 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.603084 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.603085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.603086 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:52:43.603132 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.603135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.603728 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.603729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.604293 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.604295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.604620 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.604622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.604676 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.604692 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:52:43.604917 (+ 225us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:52:43.604924 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.604965 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.605011 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 02:52:43.606217 1662 raft_consensus.cc:1261] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Refusing update from remote peer b194bddd4b734fe2a2f2896984cce6bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:43.606241 1803 raft_consensus.cc:1261] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Refusing update from remote peer b194bddd4b734fe2a2f2896984cce6bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:43.606483 2021 consensus_queue.cc:1035] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:43.606619 2042 consensus_queue.cc:1035] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:43.607218 2019 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: TxnStatusTablet state changed. Reason: New leader b194bddd4b734fe2a2f2896984cce6bc. Latest consensus state: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.607299 2019 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:43.607268 2023 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: TxnStatusTablet state changed. Reason: New leader b194bddd4b734fe2a2f2896984cce6bc. Latest consensus state: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.607362 2023 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:43.608045 2021 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.608054 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333796100529139712 to 7333796100497186816. Current Snapshot: MvccSnapshot[applied={T|T < 7333796100529139712}]
I20260927 02:52:43.608172 2021 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:52:43.608464 2023 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.608541 2023 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P 93da5f502f06428b865c05b361f78435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:43.608985 2042 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.609054 2042 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P b194bddd4b734fe2a2f2896984cce6bc: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:52:43.609390 2019 tablet_replica.cc:439] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b194bddd4b734fe2a2f2896984cce6bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 } } peers { permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 } } peers { permanent_uuid: "b194bddd4b734fe2a2f2896984cce6bc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } }
I20260927 02:52:43.609462 2019 tablet_replica.cc:442] T d563245b03b940349ded32ed297b09b0 P 1b5a363a714742c2a057b33f2ab6e3b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:43.612232 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.609967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.0.98.3:36705)
0927 02:52:43.610015 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.610017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.610024 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:52:43.610039 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.610040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.610048 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.610087 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:43.610230 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.610649 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.611683 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.611872 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.611919 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.612050 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:43.612078 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:43.612113 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.612173 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.612187 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:52:43.616631 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.614176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.98.3:44733)
0927 02:52:43.614239 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:43.614250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:43.614285 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:43.614418 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:43.614419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:43.614420 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:52:43.614462 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.614463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.615325 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.615326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.615886 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:43.615888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.616308 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.616308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:43.616347 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.616368 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:52:43.616500 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:52:43.616502 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:52:43.616530 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:52:43.616556 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:52:43.616633 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.614238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44733 (local address 127.0.98.3:36705)
0927 02:52:43.614272 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.614274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.614282 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:43.614349 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.614350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.614359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.614413 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:43.614510 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.615260 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.615952 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.616243 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.616312 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.616426 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:43.616454 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:43.616498 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.616533 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.616549 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:52:43.618662 1803 raft_consensus.cc:1261] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Refusing update from remote peer b194bddd4b734fe2a2f2896984cce6bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:43.618808 1662 raft_consensus.cc:1261] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Refusing update from remote peer b194bddd4b734fe2a2f2896984cce6bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:43.618876 2021 consensus_queue.cc:1035] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "93da5f502f06428b865c05b361f78435" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:43.619050 2021 consensus_queue.cc:1035] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b5a363a714742c2a057b33f2ab6e3b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:43.632498 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:43.632588 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:43.633133 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c0055fc967f4f0690d431dc7b611ef3: Sending DeleteTablet for 3 replicas of tablet 99a9f10dbb86466cb88a6c452c5fadff
I20260927 02:52:43.633334 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a9f10dbb86466cb88a6c452c5fadff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41488
I20260927 02:52:43.633373 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a9f10dbb86466cb88a6c452c5fadff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38332
I20260927 02:52:43.633360 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a9f10dbb86466cb88a6c452c5fadff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36376
I20260927 02:52:43.633445 2037 tablet_replica.cc:330] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: stopping tablet replica
I20260927 02:52:43.633455 2036 tablet_replica.cc:330] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: stopping tablet replica
I20260927 02:52:43.633468 2035 tablet_replica.cc:330] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: stopping tablet replica
I20260927 02:52:43.633486 2037 raft_consensus.cc:2229] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:43.633505 2036 raft_consensus.cc:2229] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:43.633515 2037 raft_consensus.cc:2258] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:43.633512 2035 raft_consensus.cc:2229] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:52:43.633535 2036 raft_consensus.cc:2258] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:43.633585 2035 raft_consensus.cc:2258] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:43.633862 2035 ts_tablet_manager.cc:1905] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:43.634282 2037 ts_tablet_manager.cc:1905] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:43.635427 2035 ts_tablet_manager.cc:1918] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:43.635471 2035 log.cc:1196] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/99a9f10dbb86466cb88a6c452c5fadff
I20260927 02:52:43.635658 2035 ts_tablet_manager.cc:1939] T 99a9f10dbb86466cb88a6c452c5fadff P b194bddd4b734fe2a2f2896984cce6bc: Deleting consensus metadata
I20260927 02:52:43.635661 2037 ts_tablet_manager.cc:1918] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:43.635699 2037 log.cc:1196] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/99a9f10dbb86466cb88a6c452c5fadff
I20260927 02:52:43.635875 2037 ts_tablet_manager.cc:1939] T 99a9f10dbb86466cb88a6c452c5fadff P 1b5a363a714742c2a057b33f2ab6e3b0: Deleting consensus metadata
I20260927 02:52:43.635897 2036 ts_tablet_manager.cc:1905] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:43.636233 1488 catalog_manager.cc:4841] TS 1b5a363a714742c2a057b33f2ab6e3b0 (127.0.98.1:39535): tablet 99a9f10dbb86466cb88a6c452c5fadff (table test-table [id=0f506d7f9c864e9eb687bf71825b3fb4]) successfully deleted
I20260927 02:52:43.637116 2036 ts_tablet_manager.cc:1918] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:43.637172 2036 log.cc:1196] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/99a9f10dbb86466cb88a6c452c5fadff
I20260927 02:52:43.637188 1489 catalog_manager.cc:4841] TS b194bddd4b734fe2a2f2896984cce6bc (127.0.98.3:36705): tablet 99a9f10dbb86466cb88a6c452c5fadff (table test-table [id=0f506d7f9c864e9eb687bf71825b3fb4]) successfully deleted
I20260927 02:52:43.637349 2036 ts_tablet_manager.cc:1939] T 99a9f10dbb86466cb88a6c452c5fadff P 93da5f502f06428b865c05b361f78435: Deleting consensus metadata
I20260927 02:52:43.637595 1488 catalog_manager.cc:4841] TS 93da5f502f06428b865c05b361f78435 (127.0.98.2:41769): tablet 99a9f10dbb86466cb88a6c452c5fadff (table test-table [id=0f506d7f9c864e9eb687bf71825b3fb4]) successfully deleted
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:52:43.647030 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.636032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.0.98.1:39535)
0927 02:52:43.636995 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.636997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.638913 (+ 1916us) server_negotiation.cc:407] Connection header received
0927 02:52:43.638974 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.638975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.638986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.639031 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:43.639793 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.640328 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.641403 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.641621 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.641672 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.644604 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:43.644613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:43.644613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:43.644618 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:43.646005 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.646297 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.646301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.646301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.646349 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.646512 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.646513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.646513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.646633 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:43.646777 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.646849 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.646924 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943}
W20260927 02:52:43.647223 1684 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38346
I20260927 02:52:43.652940 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:43.648008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.0.98.62:46767)
0927 02:52:43.648053 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:43.648055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:43.648063 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:43.648105 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:43.648106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:43.648115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:43.648154 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:43.648576 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.649033 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.649654 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:43.649850 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:43.649901 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:43.650861 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:43.650862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:43.650862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:43.650869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:43.652026 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.652315 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.652316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.652316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.652376 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:43.652526 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:43.652527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:43.652527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:43.652627 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:43.652713 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:43.652791 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:52:43.652857 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 02:52:43.653128 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54912
I20260927 02:52:43.653669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 1576
I20260927 02:52:43.658207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 1717
I20260927 02:52:43.663341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 1858
I20260927 02:52:43.667400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 1472
2026-09-27T02:52:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6562 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[2072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[2072](info): set up 2 sockets
Sep 27 02:52:43 dist-test-slave-mfx9 krb5kdc[2072](info): commencing operation
krb5kdc: starting...
W20260927 02:52:44.614236 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:52:44 dist-test-slave-mfx9 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:52:44.722784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:44.800400 2085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:44.800622 2085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:44.800643 2085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:44.801828 2085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:44.801872 2085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:52:44.801884 2085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:44.801895 2085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:44.801906 2085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:44.801919 2085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:44.803392 2085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46827
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:44.803613 2085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:44.803845 2085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:44.808501 2091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:44.808624 2085 server_base.cc:1034] running on GCE node
W20260927 02:52:44.808518 2090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:44.809931 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:44.810184 2085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:44.810462 2085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:44.811606 2085 hybrid_clock.cc:648] HybridClock initialized: now 1790477564811586 us; error 35 us; skew 500 ppm
Sep 27 02:52:44 dist-test-slave-mfx9 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:44.814006 2085 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:44.814450 2085 webserver.cc:460] Webserver started at http://127.0.98.62:35593/ using document root <none> and password file <none>
I20260927 02:52:44.814640 2085 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:44.814685 2085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:44.814836 2085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:44.815881 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "8494b0354d0444f983d9b72b847100fd"
format_stamp: "Formatted at 2026-09-27 02:52:44 on dist-test-slave-mfx9"
server_key: "0d7e07b8eb10fd4935ad75172e0ea1ed"
server_key_iv: "f8e98da92f30a227a5cf668fd220916e"
server_key_version: "encryptionkey@0"
I20260927 02:52:44.816195 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "8494b0354d0444f983d9b72b847100fd"
format_stamp: "Formatted at 2026-09-27 02:52:44 on dist-test-slave-mfx9"
server_key: "0d7e07b8eb10fd4935ad75172e0ea1ed"
server_key_iv: "f8e98da92f30a227a5cf668fd220916e"
server_key_version: "encryptionkey@0"
I20260927 02:52:44.817404 2085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:52:44.818053 2100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:44.818197 2085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:44.818275 2085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "8494b0354d0444f983d9b72b847100fd"
format_stamp: "Formatted at 2026-09-27 02:52:44 on dist-test-slave-mfx9"
server_key: "0d7e07b8eb10fd4935ad75172e0ea1ed"
server_key_iv: "f8e98da92f30a227a5cf668fd220916e"
server_key_version: "encryptionkey@0"
I20260927 02:52:44.818333 2085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:44.835843 2085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:44.838024 2085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:44.838162 2085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:44.842479 2085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46827
I20260927 02:52:44.842532 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46827 every 8 connection(s)
I20260927 02:52:44.842850 2085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:44.843442 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:44.845613 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: Bootstrap starting.
I20260927 02:52:44.846194 2163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:44.846455 2163 log.cc:824] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:44.846967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 2085
I20260927 02:52:44.847064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:44.847154 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: No bootstrap required, opened a new log
I20260927 02:52:44.847224 392 external_mini_cluster.cc:1428] Setting key 27542d92c13ad7631f875f3d04248bc7
I20260927 02:52:44.849344 2163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8494b0354d0444f983d9b72b847100fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260927 02:52:44.849483 2163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:44.849512 2163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8494b0354d0444f983d9b72b847100fd, State: Initialized, Role: FOLLOWER
I20260927 02:52:44.849653 2163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8494b0354d0444f983d9b72b847100fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260927 02:52:44.849720 2163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:44.849749 2163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:44.849797 2163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:44.850289 2163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8494b0354d0444f983d9b72b847100fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260927 02:52:44.850386 2163 leader_election.cc:302] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8494b0354d0444f983d9b72b847100fd; no voters:
I20260927 02:52:44.850551 2163 leader_election.cc:288] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:44.850595 2168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:44.850740 2168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [term 1 LEADER]: Becoming Leader. State: Replica: 8494b0354d0444f983d9b72b847100fd, State: Running, Role: LEADER
I20260927 02:52:44.850824 2163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:44.850915 2168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8494b0354d0444f983d9b72b847100fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260927 02:52:44.852073 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8494b0354d0444f983d9b72b847100fd. Latest consensus state: current_term: 1 leader_uuid: "8494b0354d0444f983d9b72b847100fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8494b0354d0444f983d9b72b847100fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } } }
I20260927 02:52:44.852185 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8494b0354d0444f983d9b72b847100fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8494b0354d0444f983d9b72b847100fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } } }
W20260927 02:52:44.852483 2183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:52:44.852535 2183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:52:44.852491 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:44.852499 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:44.852705 2185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:44.852810 2185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:44.854162 2185 catalog_manager.cc:1347] Generated new cluster ID: 0fca379d8bf549e593b2c43b7112deea
I20260927 02:52:44.854213 2185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:52:44 dist-test-slave-mfx9 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:44.858279 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:44.847761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.0.98.62:46827)
0927 02:52:44.847932 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:44.847942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:44.847974 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:52:44.849314 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:44.849321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:44.849382 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:44.849511 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:44.850166 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:44.851444 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:44.852473 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:44.853058 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:44.853162 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:44.855430 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:44.855434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:44.855434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:44.855446 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:44.856940 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:44.857241 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:44.857243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:44.857244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:44.857349 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:44.857529 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:44.857530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:44.857531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:44.857755 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:44.857900 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:44.857992 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:52:44.858121 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260927 02:52:44.882941 2185 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:44.883474 2185 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:44.899260 2185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: Generated new TSK 0
I20260927 02:52:44.899446 2185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:52:44.925563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:45.026677 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:45.026949 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:45.026988 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:45.028344 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:45.028388 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:45.028416 2189 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:52:45.028432 2189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:52:45.028461 2189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:45.029968 2189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:45.030203 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:45.030441 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:45.035468 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:45.035531 2189 server_base.cc:1034] running on GCE node
W20260927 02:52:45.035488 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:45.035512 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:45.035917 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:45.036135 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:45.037268 2189 hybrid_clock.cc:648] HybridClock initialized: now 1790477565037256 us; error 29 us; skew 500 ppm
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:45.039669 2189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:52:45.040104 2189 webserver.cc:460] Webserver started at http://127.0.98.1:35425/ using document root <none> and password file <none>
I20260927 02:52:45.040297 2189 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:45.040344 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:45.040483 2189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:45.041540 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "6dbccc61d945442a9cccc5c6950ece27"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "7c1d6a42ea97efecfd03e8d26f00286e"
server_key_iv: "9c867fc639ff9fa57a3670945caa367b"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.041893 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6dbccc61d945442a9cccc5c6950ece27"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "7c1d6a42ea97efecfd03e8d26f00286e"
server_key_iv: "9c867fc639ff9fa57a3670945caa367b"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.043143 2189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:45.043840 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.044008 2189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:45.044087 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "6dbccc61d945442a9cccc5c6950ece27"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "7c1d6a42ea97efecfd03e8d26f00286e"
server_key_iv: "9c867fc639ff9fa57a3670945caa367b"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.044142 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:45.052914 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:45.054972 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:45.055104 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:45.055553 2189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:45.055593 2189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.055622 2189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:45.055646 2189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:45.061977 2189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44181
I20260927 02:52:45.062067 2319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44181 every 8 connection(s)
I20260927 02:52:45.062322 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:52:45.065042 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.056110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49727 (local address 127.0.98.62:46827)
0927 02:52:45.056170 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.056173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.056372 (+ 199us) server_negotiation.cc:407] Connection header received
0927 02:52:45.056978 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.056980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.056999 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.057071 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:45.057968 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.058780 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.059758 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.059980 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.060037 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.062266 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:45.062268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:45.062269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:45.062279 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:45.063820 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.064177 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.064178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.064178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.064232 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.064441 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.064442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.064442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.064553 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:45.064607 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.064855 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.064949 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:52:45.065196 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.056075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.1:49727)
0927 02:52:45.056253 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.056290 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.056866 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.057172 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.057178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.057535 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:45.057853 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.057858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.058881 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.058882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.059675 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.059678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.060081 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.060082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.060216 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.060717 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:45.060765 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:45.062171 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:45.063894 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.063900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.063903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.064108 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.064284 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.064285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.064285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.064388 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.064603 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:45.064609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:45.064742 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.064828 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.064978 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260927 02:52:45.065662 2320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260927 02:52:45.065739 2320 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:45.065902 2320 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260927 02:52:45.066403 2116 ts_manager.cc:194] Registered new tserver with Master: 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1:44181)
I20260927 02:52:45.067256 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49727
I20260927 02:52:45.070318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 2189
I20260927 02:52:45.070425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:45.070580 392 external_mini_cluster.cc:1428] Setting key 56374068c0bdc5c6d729c2f8452a0244
I20260927 02:52:45.071419 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.066575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.1:54865)
0927 02:52:45.066845 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.066856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.066912 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.067015 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.067015 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.067186 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:45.067242 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.067243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.067912 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.067913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.068432 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.068433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.068723 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.068724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.068774 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.069092 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:45.069095 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:45.069335 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:45.070511 (+ 1176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.070512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.070512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.070684 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.070873 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.070873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.070874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.070929 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.071149 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:45.071150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:45.071196 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.071266 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.071343 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":71,"threads_started":1}
I20260927 02:52:45.071448 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.066619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54865 (local address 127.0.98.62:46827)
0927 02:52:45.066658 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.066659 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.066901 (+ 242us) server_negotiation.cc:407] Connection header received
0927 02:52:45.066960 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.066961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.066969 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.067011 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:45.067299 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.067831 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.068493 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.068671 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.068720 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.069404 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:45.069405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:45.069405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:45.069411 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:45.070431 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.070750 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.070751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.070751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.070823 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.070979 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.070979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.070980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.071081 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:45.071143 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.071299 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.071371 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:52:45.093385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:45.172936 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:45.173126 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:45.173157 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:45.174257 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:45.174293 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:45.174319 2329 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:52:45.174335 2329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:52:45.174363 2329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:45.175827 2329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:45.176052 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:45.176242 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:45.180938 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:45.181015 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:45.180998 2329 server_base.cc:1034] running on GCE node
W20260927 02:52:45.180910 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:45.181270 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:45.181475 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:45.182606 2329 hybrid_clock.cc:648] HybridClock initialized: now 1790477565182571 us; error 53 us; skew 500 ppm
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:45.188675 2329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:52:45.189095 2329 webserver.cc:460] Webserver started at http://127.0.98.2:38515/ using document root <none> and password file <none>
I20260927 02:52:45.189266 2329 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:45.189304 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:45.189395 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:45.190387 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "f07866aa5f3c48c7883e9bcc3d300f48"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "b43e063b6cc7753f46cbf4ece4d5d72f"
server_key_iv: "370427a115764aee4cb5724554bd8794"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.190670 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f07866aa5f3c48c7883e9bcc3d300f48"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "b43e063b6cc7753f46cbf4ece4d5d72f"
server_key_iv: "370427a115764aee4cb5724554bd8794"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.191891 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:45.192507 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.192641 2329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:45.192718 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "f07866aa5f3c48c7883e9bcc3d300f48"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "b43e063b6cc7753f46cbf4ece4d5d72f"
server_key_iv: "370427a115764aee4cb5724554bd8794"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.192783 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:45.205042 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:45.207311 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:45.207448 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:45.207985 2329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:45.208022 2329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.208043 2329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:45.208055 2329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:45.214643 2329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45427
I20260927 02:52:45.214690 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45427 every 8 connection(s)
I20260927 02:52:45.215054 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:52:45.217041 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.208431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43709 (local address 127.0.98.62:46827)
0927 02:52:45.208498 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.208501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.208660 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:52:45.209108 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.209109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.209121 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.209196 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:45.209961 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.210722 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.211782 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.211962 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.212015 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.214270 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:45.214272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:45.214272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:45.214279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:45.215856 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.216197 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.216198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.216199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.216254 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.216468 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.216469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.216469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.216578 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:45.216632 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.216883 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.216955 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:52:45.217190 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.208382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.2:43709)
0927 02:52:45.208538 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.208579 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.209023 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.209272 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.209275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.209582 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:45.209855 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.209861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.210876 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.210877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.211700 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.211702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.212036 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.212037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.212153 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.212630 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:45.212642 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:45.214189 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:45.215928 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.215933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.215935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.216132 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.216304 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.216305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.216305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.216410 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.216628 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:45.216634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:45.216762 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.216846 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.216992 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260927 02:52:45.217558 2460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260927 02:52:45.217648 2460 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:45.217833 2460 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260927 02:52:45.217798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 2329
I20260927 02:52:45.217928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:45.218093 392 external_mini_cluster.cc:1428] Setting key 9e142c1146ed5f156ce1dec6cefffd05
I20260927 02:52:45.218432 2116 ts_manager.cc:194] Registered new tserver with Master: f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427)
I20260927 02:52:45.218888 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43709
I20260927 02:52:45.224452 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.219108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45265 (local address 127.0.98.62:46827)
0927 02:52:45.219143 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.219144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.219309 (+ 165us) server_negotiation.cc:407] Connection header received
0927 02:52:45.219331 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.219331 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.219342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.219385 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:45.219637 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.220319 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.221074 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.221253 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.221316 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.222209 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:45.222210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:45.222211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:45.222216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:45.223467 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.223809 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.223810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.223810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.223858 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.224013 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.224013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.224013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.224131 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:45.224174 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.224331 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.224393 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:52:45.224520 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.219078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.2:45265)
0927 02:52:45.219220 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.219228 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.219274 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.219391 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.219392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.219527 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:45.219586 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.219587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.220444 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.220445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.221012 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.221014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.221331 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.221332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.221417 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.221770 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:45.221775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:45.222140 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:45.223538 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.223539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.223539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.223743 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.223898 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.223899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.223899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.223960 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.224183 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:45.224184 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:45.224227 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.224315 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.224420 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:52:45.241364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:45.319743 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:45.319967 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:45.319990 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:45.321148 2469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:45.321197 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:45.321218 2469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:52:45.321234 2469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:52:45.321249 2469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:45.322659 2469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:45.322898 2469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:45.323093 2469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:45.327888 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:45.327965 2474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:45.328017 2469 server_base.cc:1034] running on GCE node
W20260927 02:52:45.327965 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:45.328332 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:45.328550 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:45.329684 2469 hybrid_clock.cc:648] HybridClock initialized: now 1790477565329671 us; error 27 us; skew 500 ppm
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:45.332080 2469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:52:45.332494 2469 webserver.cc:460] Webserver started at http://127.0.98.3:33311/ using document root <none> and password file <none>
I20260927 02:52:45.332684 2469 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:45.332731 2469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:45.332852 2469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:45.333859 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "eded432842a44073afcaf37cbcfdbb86"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "4cc5942b04171a6e7362caf2e8fe2086"
server_key_iv: "6e931843f219e3e9c9ad96df7557a75a"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.334183 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eded432842a44073afcaf37cbcfdbb86"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "4cc5942b04171a6e7362caf2e8fe2086"
server_key_iv: "6e931843f219e3e9c9ad96df7557a75a"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.335459 2469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:45.336130 2484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.336274 2469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:45.336360 2469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "eded432842a44073afcaf37cbcfdbb86"
format_stamp: "Formatted at 2026-09-27 02:52:45 on dist-test-slave-mfx9"
server_key: "4cc5942b04171a6e7362caf2e8fe2086"
server_key_iv: "6e931843f219e3e9c9ad96df7557a75a"
server_key_version: "encryptionkey@0"
I20260927 02:52:45.336426 2469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:45.349539 2469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:45.351611 2469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:45.351738 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:45.352313 2469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:45.352370 2469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.352408 2469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:45.352432 2469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.358697 2469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41119
I20260927 02:52:45.358871 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41119 every 8 connection(s)
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:45.359112 2469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:52:45.361624 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.352773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58015 (local address 127.0.98.62:46827)
0927 02:52:45.352837 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.352840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.353030 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:52:45.353521 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.353522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.353535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.353612 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:45.354371 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.355145 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.356227 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.356525 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.356593 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.359512 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:45.359514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:45.359514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:45.359523 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:45.360584 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.360939 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.360940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.360941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.360995 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.361167 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.361168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.361168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.361262 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:45.361308 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.361500 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.361562 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:52:45.361789 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.352760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.3:58015)
0927 02:52:45.352906 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.352945 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.353420 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.353674 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.353678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.354008 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:45.354275 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.354283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.355239 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.355240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.356147 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.356150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.356623 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.356624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.356782 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.357745 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:45.357756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:45.359428 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:45.360648 (+ 1220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.360653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.360654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.360862 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.361038 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.361039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.361039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.361127 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.361298 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:45.361301 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:45.361411 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.361481 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.361593 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260927 02:52:45.362210 2600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260927 02:52:45.362289 2600 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:45.362453 2600 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260927 02:52:45.362788 2117 ts_manager.cc:194] Registered new tserver with Master: eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:45.363267 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58015
I20260927 02:52:45.365872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 2469
I20260927 02:52:45.365964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:45.366120 392 external_mini_cluster.cc:1428] Setting key 66efbe012e3d30445948e0d8c2d40aac
I20260927 02:52:45.367125 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:52:45.368465 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.363252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44485 (local address 127.0.98.62:46827)
0927 02:52:45.363289 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.363291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.363342 (+ 51us) server_negotiation.cc:407] Connection header received
0927 02:52:45.363408 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.363409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.363418 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.363461 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:45.363729 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.364186 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.364835 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.365032 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.365082 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.365977 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:45.365979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:45.365979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:45.365987 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:45.367465 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.367849 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.367850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.367850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.367907 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.368083 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.368083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.368083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.368185 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:45.368235 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.368345 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.368400 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:52:45.368525 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.363166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.3:44485)
0927 02:52:45.363285 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.363296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.363353 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.363464 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.363465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.363624 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:45.363668 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.363670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.364266 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.364267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.364769 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.364771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.365087 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.365088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.365158 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.365536 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:45.365541 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:45.365872 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:45.367538 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.367539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.367546 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.367777 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.367971 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.367971 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.367972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.368031 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.368233 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:45.368234 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:45.368272 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.368331 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.368426 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:45.387347 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.381504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.0.98.62:46827)
0927 02:52:45.381551 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.381552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.381559 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:52:45.381581 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.381582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.381590 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.381641 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:45.382141 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.382630 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.383387 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.383574 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.383621 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.385527 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:45.385529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:45.385529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:45.385536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:45.386552 (+ 1016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.386846 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.386846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.386847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.386892 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.387015 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.387016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.387016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.387101 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:45.387164 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.387230 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.387284 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:52:45.388767 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:52:45.390255 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:45.396237 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.392125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41119 (local address 127.0.0.1:34706)
0927 02:52:45.392415 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.392425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.392483 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.392770 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.392773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.392775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:45.393049 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.393051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.393862 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.393863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.395044 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.395046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.395944 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.395945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.396016 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.396030 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.396098 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.396148 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":53,"threads_started":1}
I20260927 02:52:45.396248 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.392451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.0.98.3:41119)
0927 02:52:45.392619 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.392623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.392632 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:52:45.392671 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.392673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.392689 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.392761 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:45.393102 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.393789 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.395119 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.395872 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.395932 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.395986 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.396117 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.396156 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260927 02:52:45.396422 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.392462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.0.98.1:44181)
0927 02:52:45.392739 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.392743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.392751 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:45.392784 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.392786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.392798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.392859 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:45.392996 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.394090 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.395155 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.395861 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.395929 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.395987 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.396288 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.396327 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":77,"threads_started":1}
I20260927 02:52:45.396458 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.392205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.0.98.2:45427)
0927 02:52:45.392387 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.392392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.392403 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:52:45.392700 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.392703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.392717 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.392783 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:45.392987 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.394086 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.395490 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.395956 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.396017 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.396071 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.396238 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.396276 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260927 02:52:45.396255 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.392334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44181 (local address 127.0.0.1:54524)
0927 02:52:45.392609 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.392616 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.392652 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.392871 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.392872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.392874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:45.392927 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.392929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.394195 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.394196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.395059 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.395062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.396019 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.396020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.396091 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.396096 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.396144 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.396180 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":18,"thread_start_us":46,"threads_started":1}
I20260927 02:52:45.396665 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.392111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45427 (local address 127.0.0.1:53686)
0927 02:52:45.392249 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.392287 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.392598 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.392842 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.392843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.392845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:45.392915 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.392917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.394165 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.394166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.395380 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.395383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.396101 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.396102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.396155 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.396158 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.396570 (+ 412us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.396609 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":30,"thread_start_us":54,"threads_started":1}
I20260927 02:52:45.397372 2534 tablet_service.cc:1467] Processing CreateTablet for tablet fc5eda36df4048edbd39ca60ba375db2 (DEFAULT_TABLE table=test-table [id=1126788abad549d4b18efd4a4e99f9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:45.397374 2254 tablet_service.cc:1467] Processing CreateTablet for tablet fc5eda36df4048edbd39ca60ba375db2 (DEFAULT_TABLE table=test-table [id=1126788abad549d4b18efd4a4e99f9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:45.397445 2394 tablet_service.cc:1467] Processing CreateTablet for tablet fc5eda36df4048edbd39ca60ba375db2 (DEFAULT_TABLE table=test-table [id=1126788abad549d4b18efd4a4e99f9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:45.397696 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc5eda36df4048edbd39ca60ba375db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.397706 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc5eda36df4048edbd39ca60ba375db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.397706 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc5eda36df4048edbd39ca60ba375db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.400583 2620 tablet_bootstrap.cc:492] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Bootstrap starting.
I20260927 02:52:45.400722 2621 tablet_bootstrap.cc:492] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Bootstrap starting.
I20260927 02:52:45.401031 2622 tablet_bootstrap.cc:492] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Bootstrap starting.
I20260927 02:52:45.401394 2620 tablet_bootstrap.cc:654] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.401521 2621 tablet_bootstrap.cc:654] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.401618 2622 tablet_bootstrap.cc:654] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.401803 2621 log.cc:824] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:45.401795 2620 log.cc:824] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:45.401985 2622 log.cc:824] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:45.402514 2620 tablet_bootstrap.cc:492] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: No bootstrap required, opened a new log
I20260927 02:52:45.402514 2621 tablet_bootstrap.cc:492] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: No bootstrap required, opened a new log
I20260927 02:52:45.402581 2620 ts_tablet_manager.cc:1397] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:52:45.402581 2621 ts_tablet_manager.cc:1397] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:52:45.403205 2622 tablet_bootstrap.cc:492] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: No bootstrap required, opened a new log
I20260927 02:52:45.403262 2622 ts_tablet_manager.cc:1397] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:52:45.404426 2621 raft_consensus.cc:348] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.404426 2620 raft_consensus.cc:348] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.404577 2621 raft_consensus.cc:374] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.404577 2620 raft_consensus.cc:374] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.404515 2622 raft_consensus.cc:348] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.404621 2622 raft_consensus.cc:374] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.404610 2621 raft_consensus.cc:729] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eded432842a44073afcaf37cbcfdbb86, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.404610 2620 raft_consensus.cc:729] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07866aa5f3c48c7883e9bcc3d300f48, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.404640 2622 raft_consensus.cc:729] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.404722 2622 consensus_queue.cc:260] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.404723 2621 consensus_queue.cc:260] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.404723 2620 consensus_queue.cc:260] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.404968 2621 ts_tablet_manager.cc:1428] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:52:45.404968 2620 ts_tablet_manager.cc:1428] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:52:45.404987 2320 heartbeater.cc:503] Master 127.0.98.62:46827 was elected leader, sending a full tablet report...
I20260927 02:52:45.404968 2622 ts_tablet_manager.cc:1428] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:52:45.405069 2600 heartbeater.cc:503] Master 127.0.98.62:46827 was elected leader, sending a full tablet report...
I20260927 02:52:45.405228 2460 heartbeater.cc:503] Master 127.0.98.62:46827 was elected leader, sending a full tablet report...
W20260927 02:52:45.465569 2461 tablet.cc:2369] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:52:45.563004 2321 tablet.cc:2369] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:52:45.586802 2626 raft_consensus.cc:482] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:45.586922 2626 raft_consensus.cc:504] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.587263 2626 leader_election.cc:288] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:45.590051 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.587505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41987 (local address 127.0.98.2:45427)
0927 02:52:45.587559 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.587561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.587606 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:52:45.587678 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.587679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.587691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.587746 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:45.587910 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.588410 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.589357 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.589755 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.589803 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.589812 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.589949 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.589986 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:52:45.590104 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.587612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33835 (local address 127.0.98.3:41119)
0927 02:52:45.587647 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.587649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.587741 (+ 92us) server_negotiation.cc:407] Connection header received
0927 02:52:45.587784 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.587785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.587795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.587884 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:45.587996 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.588446 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.589428 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.589800 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.589879 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.589889 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.590007 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.590046 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:52:45.590098 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.587424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45427 (local address 127.0.98.1:41987)
0927 02:52:45.587531 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.587543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.587617 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.587767 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.587768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.587770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:45.587847 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.587849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.588475 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.588477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.589283 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.589285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.589818 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.589819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.589884 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.589888 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.589936 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.589975 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":26,"thread_start_us":60,"threads_started":1}
I20260927 02:52:45.590103 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.587424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41119 (local address 127.0.98.1:33835)
0927 02:52:45.587687 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.587695 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.587738 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.587889 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.587889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.587891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:45.587946 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.587947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.588515 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.588516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.589347 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.589350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.589885 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.589886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.589943 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.589947 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.589996 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.590035 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":5,"thread_start_us":59,"threads_started":1}
I20260927 02:52:45.590458 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc5eda36df4048edbd39ca60ba375db2" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" is_pre_election: true
I20260927 02:52:45.590458 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc5eda36df4048edbd39ca60ba375db2" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86" is_pre_election: true
I20260927 02:52:45.590605 2414 raft_consensus.cc:2454] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:45.590605 2554 raft_consensus.cc:2454] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:45.590821 2205 leader_election.cc:302] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, f07866aa5f3c48c7883e9bcc3d300f48; no voters:
I20260927 02:52:45.591022 2626 raft_consensus.cc:2781] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:45.591068 2626 raft_consensus.cc:482] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:45.591082 2626 raft_consensus.cc:3037] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.591760 2626 raft_consensus.cc:504] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.591884 2626 leader_election.cc:288] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Requested vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:45.592026 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc5eda36df4048edbd39ca60ba375db2" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48"
I20260927 02:52:45.592087 2414 raft_consensus.cc:3037] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.592720 2414 raft_consensus.cc:2454] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:45.592816 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc5eda36df4048edbd39ca60ba375db2" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86"
I20260927 02:52:45.592862 2205 leader_election.cc:302] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, f07866aa5f3c48c7883e9bcc3d300f48; no voters:
I20260927 02:52:45.592901 2554 raft_consensus.cc:3037] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.592937 2626 raft_consensus.cc:2781] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:45.593053 2626 raft_consensus.cc:686] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Running, Role: LEADER
I20260927 02:52:45.593184 2626 consensus_queue.cc:237] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.593608 2554 raft_consensus.cc:2454] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:45.593958 2115 catalog_manager.cc:5495] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } health_report { overall_health: UNKNOWN } } }
W20260927 02:52:45.609663 2601 tablet.cc:2369] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:52:45 dist-test-slave-mfx9 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:45.641449 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.635490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.0.98.62:46827)
0927 02:52:45.635535 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.635537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.635643 (+ 106us) server_negotiation.cc:407] Connection header received
0927 02:52:45.635690 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.635691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.635699 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.635752 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:45.636179 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.636664 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.637313 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.637509 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.637554 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.639464 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:45.639465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:45.639466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:45.639472 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:45.640596 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.640894 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.640895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.640895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.640942 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:45.641071 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:45.641072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:45.641072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:45.641161 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:45.641205 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.641309 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.641384 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:52:45.641532 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.635441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.0.1:50468)
0927 02:52:45.635583 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.635592 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.635649 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.635765 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.635765 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.636076 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:45.636124 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.636126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.636719 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.636719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.637260 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.637261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.637558 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.637558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.637594 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.638068 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:45.638072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:45.639386 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:45.640658 (+ 1272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.640664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.640666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.640841 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.640986 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:45.640986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:45.640986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:45.641034 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:45.641200 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:45.641205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:45.641240 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.641294 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.641440 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":68,"thread_start_us":64,"threads_started":1}
I20260927 02:52:45.643204 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50468:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:52:45.643380 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:45.644650 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 9f399797d3394eb5a85fb2b06fd9d37b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a920ea930b494500a5de3ce6f4b140ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:52:45.644649 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 9f399797d3394eb5a85fb2b06fd9d37b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a920ea930b494500a5de3ce6f4b140ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:52:45.644651 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 9f399797d3394eb5a85fb2b06fd9d37b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a920ea930b494500a5de3ce6f4b140ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:52:45.644802 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f399797d3394eb5a85fb2b06fd9d37b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.644802 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f399797d3394eb5a85fb2b06fd9d37b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.644804 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f399797d3394eb5a85fb2b06fd9d37b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.645875 2620 tablet_bootstrap.cc:492] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: Bootstrap starting.
I20260927 02:52:45.646238 2621 tablet_bootstrap.cc:492] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: Bootstrap starting.
I20260927 02:52:45.646481 2620 tablet_bootstrap.cc:654] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.646808 2621 tablet_bootstrap.cc:654] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.647149 2622 tablet_bootstrap.cc:492] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: Bootstrap starting.
I20260927 02:52:45.647163 2620 tablet_bootstrap.cc:492] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: No bootstrap required, opened a new log
I20260927 02:52:45.647203 2620 ts_tablet_manager.cc:1397] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:45.647341 2620 raft_consensus.cc:348] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.647424 2620 raft_consensus.cc:374] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.647451 2620 raft_consensus.cc:729] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07866aa5f3c48c7883e9bcc3d300f48, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.647490 2620 consensus_queue.cc:260] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.647555 2622 tablet_bootstrap.cc:654] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.647571 2620 ts_tablet_manager.cc:1428] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.647922 2628 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.648013 2628 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:45.648231 2622 tablet_bootstrap.cc:492] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: No bootstrap required, opened a new log
I20260927 02:52:45.648274 2622 ts_tablet_manager.cc:1397] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:45.648556 2622 raft_consensus.cc:348] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.648619 2622 raft_consensus.cc:374] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.648638 2622 raft_consensus.cc:729] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.648679 2622 consensus_queue.cc:260] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.648782 2622 ts_tablet_manager.cc:1428] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.648824 2626 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.648917 2626 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:45.648980 2621 tablet_bootstrap.cc:492] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: No bootstrap required, opened a new log
I20260927 02:52:45.649029 2621 ts_tablet_manager.cc:1397] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 02:52:45.649158 2621 raft_consensus.cc:348] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.649209 2621 raft_consensus.cc:374] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.649226 2621 raft_consensus.cc:729] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eded432842a44073afcaf37cbcfdbb86, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.649263 2621 consensus_queue.cc:260] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.649343 2621 ts_tablet_manager.cc:1428] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.649427 2627 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.649515 2627 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:45.893328 2626 raft_consensus.cc:482] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:45.893419 2626 raft_consensus.cc:504] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.893567 2626 leader_election.cc:288] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:45.893774 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f399797d3394eb5a85fb2b06fd9d37b" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" is_pre_election: true
I20260927 02:52:45.893793 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f399797d3394eb5a85fb2b06fd9d37b" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86" is_pre_election: true
I20260927 02:52:45.893867 2554 raft_consensus.cc:2454] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:45.893862 2414 raft_consensus.cc:2454] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:45.894023 2208 leader_election.cc:302] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, eded432842a44073afcaf37cbcfdbb86; no voters:
I20260927 02:52:45.894114 2626 raft_consensus.cc:2781] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:45.894150 2626 raft_consensus.cc:482] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:45.894166 2626 raft_consensus.cc:3037] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.894847 2626 raft_consensus.cc:504] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.894953 2626 leader_election.cc:288] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Requested vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:45.895108 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f399797d3394eb5a85fb2b06fd9d37b" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48"
I20260927 02:52:45.895123 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f399797d3394eb5a85fb2b06fd9d37b" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86"
I20260927 02:52:45.895185 2414 raft_consensus.cc:3037] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.895185 2554 raft_consensus.cc:3037] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.895889 2554 raft_consensus.cc:2454] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:45.895895 2414 raft_consensus.cc:2454] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:45.896056 2205 leader_election.cc:302] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, f07866aa5f3c48c7883e9bcc3d300f48; no voters:
I20260927 02:52:45.896173 2626 raft_consensus.cc:2781] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:45.896236 2626 raft_consensus.cc:686] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Running, Role: LEADER
I20260927 02:52:45.896291 2626 consensus_queue.cc:237] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.896574 2631 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: TxnStatusTablet state changed. Reason: New leader 6dbccc61d945442a9cccc5c6950ece27. Latest consensus state: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.896672 2631 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:52:45.896883 2115 catalog_manager.cc:5495] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } health_report { overall_health: UNKNOWN } } }
I20260927 02:52:45.897171 2643 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:52:45.897306 2643 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:52:45.930058 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.927591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.0.98.1:44181)
0927 02:52:45.927707 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.927709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.927716 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:52:45.927755 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.927756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.927776 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.927845 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:45.928001 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.928504 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.929236 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.929461 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.929515 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.929697 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:45.929803 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:45.929872 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.929936 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.929958 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260927 02:52:45.930048 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.927499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44181 (local address 127.0.0.1:41590)
0927 02:52:45.927569 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.927580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.927623 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.927871 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.927873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.927874 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:52:45.927926 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.927928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.928589 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.928590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.929161 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.929163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.929530 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.929531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.929597 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.929611 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:52:45.929864 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:52:45.929871 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.929927 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.929979 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:52:45.931870 2414 raft_consensus.cc:1261] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:45.931900 2554 raft_consensus.cc:1261] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:45.932274 2631 consensus_queue.cc:1035] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:45.932406 2626 consensus_queue.cc:1035] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:45.933457 2628 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: TxnStatusTablet state changed. Reason: New leader 6dbccc61d945442a9cccc5c6950ece27. Latest consensus state: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.933544 2628 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:45.933514 2627 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: TxnStatusTablet state changed. Reason: New leader 6dbccc61d945442a9cccc5c6950ece27. Latest consensus state: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.933609 2627 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:45.934574 2631 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.934679 2631 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:52:45.934820 2627 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.934890 2627 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P eded432842a44073afcaf37cbcfdbb86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:45.934986 2649 mvcc.cc:205] Tried to move back new op lower bound from 7333796110054400000 to 7333796109911465984. Current Snapshot: MvccSnapshot[applied={T|T < 7333796110054400000}]
I20260927 02:52:45.935289 2628 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.935361 2628 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P f07866aa5f3c48c7883e9bcc3d300f48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:52:45.934595 2626 tablet_replica.cc:439] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } }
I20260927 02:52:45.935626 2626 tablet_replica.cc:442] T 9f399797d3394eb5a85fb2b06fd9d37b P 6dbccc61d945442a9cccc5c6950ece27: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:52:45.935709 2648 mvcc.cc:205] Tried to move back new op lower bound from 7333796110054400000 to 7333796109911465984. Current Snapshot: MvccSnapshot[applied={T|T < 7333796110054400000 or (T in {7333796110054400000})}]
I20260927 02:52:45.934618 2647 mvcc.cc:205] Tried to move back new op lower bound from 7333796110054400000 to 7333796109911465984. Current Snapshot: MvccSnapshot[applied={T|T < 7333796110054400000}]
I20260927 02:52:45.941073 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.937910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.0.98.1:44181)
0927 02:52:45.937951 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.937953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.938041 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:52:45.938079 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.938080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.938089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.938151 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:45.938243 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.939514 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.940368 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.940676 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.940744 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.940873 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:45.940905 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:45.940952 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.940988 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.941006 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:52:45.945853 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.943360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44181 (local address 127.0.98.1:36545)
0927 02:52:45.943496 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:45.943506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:45.943555 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:45.943660 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:45.943660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:45.943661 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:52:45.943709 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.943710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.944513 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.944513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.945097 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:45.945099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.945518 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.945519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:45.945557 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.945569 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:52:45.945716 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:52:45.945717 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:52:45.945742 (+ 25us) client_negotiation.cc:240] Negotiation successful
0927 02:52:45.945772 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260927 02:52:45.945863 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:45.943408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36545 (local address 127.0.98.1:44181)
0927 02:52:45.943442 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:45.943444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:45.943557 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:52:45.943592 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:45.943593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:45.943601 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:45.943658 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:45.943744 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.944437 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.945165 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:45.945459 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:45.945521 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:45.945637 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:45.945666 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:45.945707 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:45.945754 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:52:45.945772 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:52:45.947916 2554 raft_consensus.cc:1261] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:45.947999 2414 raft_consensus.cc:1261] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:45.948129 2653 consensus_queue.cc:1035] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:45.948345 2653 consensus_queue.cc:1035] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:45.959110 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:45.959226 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:45.959930 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: Sending DeleteTablet for 3 replicas of tablet fc5eda36df4048edbd39ca60ba375db2
I20260927 02:52:45.960210 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet fc5eda36df4048edbd39ca60ba375db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34706
I20260927 02:52:45.960282 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet fc5eda36df4048edbd39ca60ba375db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54524
I20260927 02:52:45.960315 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet fc5eda36df4048edbd39ca60ba375db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53686
I20260927 02:52:45.960551 2666 tablet_replica.cc:330] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: stopping tablet replica
I20260927 02:52:45.960675 2664 tablet_replica.cc:330] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: stopping tablet replica
I20260927 02:52:45.960788 2664 raft_consensus.cc:2229] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:45.960738 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:52:45.960844 2664 raft_consensus.cc:2258] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:52:45.960856 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:45.961134 2666 raft_consensus.cc:2229] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:45.961247 2666 raft_consensus.cc:2258] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:45.961309 2664 ts_tablet_manager.cc:1905] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:45.961653 2666 ts_tablet_manager.cc:1905] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:45.962234 2534 tablet_service.cc:1467] Processing CreateTablet for tablet a7ed2d272dd74155b5758ca73790f22c (DEFAULT_TABLE table=test-table [id=9c7def4b376c4eefb22127cc91436360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:45.962260 2254 tablet_service.cc:1467] Processing CreateTablet for tablet a7ed2d272dd74155b5758ca73790f22c (DEFAULT_TABLE table=test-table [id=9c7def4b376c4eefb22127cc91436360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:45.962303 2394 tablet_service.cc:1467] Processing CreateTablet for tablet a7ed2d272dd74155b5758ca73790f22c (DEFAULT_TABLE table=test-table [id=9c7def4b376c4eefb22127cc91436360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:45.962337 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ed2d272dd74155b5758ca73790f22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.962409 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ed2d272dd74155b5758ca73790f22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.963608 2621 tablet_bootstrap.cc:492] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: Bootstrap starting.
I20260927 02:52:45.963701 2620 tablet_bootstrap.cc:492] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: Bootstrap starting.
I20260927 02:52:45.963971 2621 tablet_bootstrap.cc:654] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.964066 2620 tablet_bootstrap.cc:654] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.964596 2621 tablet_bootstrap.cc:492] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: No bootstrap required, opened a new log
I20260927 02:52:45.964643 2621 ts_tablet_manager.cc:1397] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:45.964701 2664 ts_tablet_manager.cc:1918] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:45.964761 2664 log.cc:1196] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/fc5eda36df4048edbd39ca60ba375db2
I20260927 02:52:45.964785 2621 raft_consensus.cc:348] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.964854 2621 raft_consensus.cc:374] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.964876 2621 raft_consensus.cc:729] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eded432842a44073afcaf37cbcfdbb86, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.964912 2621 consensus_queue.cc:260] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.964984 2621 ts_tablet_manager.cc:1428] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:45.965039 2664 ts_tablet_manager.cc:1939] T fc5eda36df4048edbd39ca60ba375db2 P eded432842a44073afcaf37cbcfdbb86: Deleting consensus metadata
I20260927 02:52:45.965161 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ed2d272dd74155b5758ca73790f22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:45.965586 2104 catalog_manager.cc:4841] TS eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119): tablet fc5eda36df4048edbd39ca60ba375db2 (table test-table [id=1126788abad549d4b18efd4a4e99f9cd]) successfully deleted
I20260927 02:52:45.966241 2622 tablet_bootstrap.cc:492] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: Bootstrap starting.
I20260927 02:52:45.966394 2666 ts_tablet_manager.cc:1918] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:45.966454 2666 log.cc:1196] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/fc5eda36df4048edbd39ca60ba375db2
I20260927 02:52:45.966638 2622 tablet_bootstrap.cc:654] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:45.966655 2666 ts_tablet_manager.cc:1939] T fc5eda36df4048edbd39ca60ba375db2 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting consensus metadata
I20260927 02:52:45.966931 2665 tablet_replica.cc:330] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: stopping tablet replica
I20260927 02:52:45.967023 2665 raft_consensus.cc:2229] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:52:45.967007 2101 catalog_manager.cc:4841] TS f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427): tablet fc5eda36df4048edbd39ca60ba375db2 (table test-table [id=1126788abad549d4b18efd4a4e99f9cd]) successfully deleted
I20260927 02:52:45.967124 2665 raft_consensus.cc:2258] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:45.967446 2665 ts_tablet_manager.cc:1905] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:45.967482 2622 tablet_bootstrap.cc:492] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: No bootstrap required, opened a new log
I20260927 02:52:45.967574 2622 ts_tablet_manager.cc:1397] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:45.967676 2622 raft_consensus.cc:348] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.967736 2622 raft_consensus.cc:374] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.967753 2622 raft_consensus.cc:729] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.967787 2622 consensus_queue.cc:260] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.967866 2622 ts_tablet_manager.cc:1428] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:45.969041 2665 ts_tablet_manager.cc:1918] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:45.969115 2665 log.cc:1196] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/fc5eda36df4048edbd39ca60ba375db2
I20260927 02:52:45.969383 2665 ts_tablet_manager.cc:1939] T fc5eda36df4048edbd39ca60ba375db2 P 6dbccc61d945442a9cccc5c6950ece27: Deleting consensus metadata
I20260927 02:52:45.969722 2101 catalog_manager.cc:4841] TS 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1:44181): tablet fc5eda36df4048edbd39ca60ba375db2 (table test-table [id=1126788abad549d4b18efd4a4e99f9cd]) successfully deleted
I20260927 02:52:45.971083 2620 tablet_bootstrap.cc:492] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: No bootstrap required, opened a new log
I20260927 02:52:45.971124 2620 ts_tablet_manager.cc:1397] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260927 02:52:45.971243 2620 raft_consensus.cc:348] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.971319 2620 raft_consensus.cc:374] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:45.971341 2620 raft_consensus.cc:729] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07866aa5f3c48c7883e9bcc3d300f48, State: Initialized, Role: FOLLOWER
I20260927 02:52:45.971385 2620 consensus_queue.cc:260] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.971473 2620 ts_tablet_manager.cc:1428] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:45.992429 2626 raft_consensus.cc:482] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:45.992542 2626 raft_consensus.cc:504] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.992688 2626 leader_election.cc:288] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:45.992985 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ed2d272dd74155b5758ca73790f22c" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86" is_pre_election: true
I20260927 02:52:45.993091 2554 raft_consensus.cc:2454] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:45.993259 2208 leader_election.cc:302] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, eded432842a44073afcaf37cbcfdbb86; no voters:
I20260927 02:52:45.993345 2626 raft_consensus.cc:2781] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:45.993417 2626 raft_consensus.cc:482] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:45.993443 2626 raft_consensus.cc:3037] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.993476 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ed2d272dd74155b5758ca73790f22c" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" is_pre_election: true
I20260927 02:52:45.993557 2414 raft_consensus.cc:2454] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:45.994486 2626 raft_consensus.cc:504] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.994866 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ed2d272dd74155b5758ca73790f22c" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48"
I20260927 02:52:45.994951 2414 raft_consensus.cc:3037] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.995894 2414 raft_consensus.cc:2454] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:45.996073 2205 leader_election.cc:302] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, f07866aa5f3c48c7883e9bcc3d300f48; no voters:
I20260927 02:52:45.996161 2631 raft_consensus.cc:2781] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:45.996212 2631 raft_consensus.cc:686] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Running, Role: LEADER
I20260927 02:52:45.996256 2631 consensus_queue.cc:237] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:45.996932 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ed2d272dd74155b5758ca73790f22c" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86"
I20260927 02:52:45.997043 2554 raft_consensus.cc:3037] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:45.997885 2554 raft_consensus.cc:2454] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:45.998073 2626 leader_election.cc:288] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Requested vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:46.003059 2115 catalog_manager.cc:5495] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } health_report { overall_health: UNKNOWN } } }
I20260927 02:52:46.014271 2554 raft_consensus.cc:1261] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:46.014312 2414 raft_consensus.cc:1261] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:46.014453 2626 consensus_queue.cc:1035] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:46.014590 2631 consensus_queue.cc:1035] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:52:46.019294 2647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:52:47.536597 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:47.536688 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:47.537428 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: Sending DeleteTablet for 3 replicas of tablet a7ed2d272dd74155b5758ca73790f22c
I20260927 02:52:47.537659 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ed2d272dd74155b5758ca73790f22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54524
I20260927 02:52:47.537670 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ed2d272dd74155b5758ca73790f22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53686
I20260927 02:52:47.537699 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ed2d272dd74155b5758ca73790f22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34706
I20260927 02:52:47.537866 2694 tablet_replica.cc:330] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: stopping tablet replica
I20260927 02:52:47.537915 2694 raft_consensus.cc:2229] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:47.538017 2693 tablet_replica.cc:330] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: stopping tablet replica
I20260927 02:52:47.538090 2693 raft_consensus.cc:2229] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:47.538131 2693 raft_consensus.cc:2258] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:47.538187 2694 raft_consensus.cc:2258] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:47.538434 2693 ts_tablet_manager.cc:1905] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:47.538919 2694 ts_tablet_manager.cc:1905] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:47.539990 2693 ts_tablet_manager.cc:1918] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:52:47.540078 2693 log.cc:1196] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/a7ed2d272dd74155b5758ca73790f22c
I20260927 02:52:47.540285 2693 ts_tablet_manager.cc:1939] T a7ed2d272dd74155b5758ca73790f22c P eded432842a44073afcaf37cbcfdbb86: Deleting consensus metadata
I20260927 02:52:47.540597 2694 ts_tablet_manager.cc:1918] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:52:47.540683 2694 log.cc:1196] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/a7ed2d272dd74155b5758ca73790f22c
I20260927 02:52:47.540661 2104 catalog_manager.cc:4841] TS eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119): tablet a7ed2d272dd74155b5758ca73790f22c (table test-table [id=9c7def4b376c4eefb22127cc91436360]) successfully deleted
I20260927 02:52:47.540859 2694 ts_tablet_manager.cc:1939] T a7ed2d272dd74155b5758ca73790f22c P f07866aa5f3c48c7883e9bcc3d300f48: Deleting consensus metadata
I20260927 02:52:47.541215 2101 catalog_manager.cc:4841] TS f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427): tablet a7ed2d272dd74155b5758ca73790f22c (table test-table [id=9c7def4b376c4eefb22127cc91436360]) successfully deleted
I20260927 02:52:47.542373 2692 tablet_replica.cc:330] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: stopping tablet replica
I20260927 02:52:47.542428 2692 raft_consensus.cc:2229] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:52:47.542481 2692 raft_consensus.cc:2258] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:47.542675 2692 ts_tablet_manager.cc:1905] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:47.544045 2692 ts_tablet_manager.cc:1918] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:52:47.544103 2692 log.cc:1196] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/a7ed2d272dd74155b5758ca73790f22c
I20260927 02:52:47.544274 2692 ts_tablet_manager.cc:1939] T a7ed2d272dd74155b5758ca73790f22c P 6dbccc61d945442a9cccc5c6950ece27: Deleting consensus metadata
I20260927 02:52:47.544631 2101 catalog_manager.cc:4841] TS 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1:44181): tablet a7ed2d272dd74155b5758ca73790f22c (table test-table [id=9c7def4b376c4eefb22127cc91436360]) successfully deleted
Sep 27 02:52:47 dist-test-slave-mfx9 krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:52:47 dist-test-slave-mfx9 krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:47.559146 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:47.553097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.0.98.62:46827)
0927 02:52:47.553216 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:47.553218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:47.553239 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:52:47.553290 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:47.553291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:47.553300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:47.553356 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:47.553820 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:47.554326 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:47.555113 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:47.555308 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:47.555352 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:47.557211 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:47.557212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:47.557213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:47.557218 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:47.558334 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:47.558594 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:47.558595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:47.558595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:47.558639 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:47.558774 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:47.558775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:47.558775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:47.558870 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:47.558947 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:47.559028 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:52:47.559084 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260927 02:52:47.560350 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:52:47.560505 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:47.561667 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 9f25c338def749c39bd3bf23a3aa5088 (DEFAULT_TABLE table=test-table [id=dd20058392584d6e9885892207c072e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:47.561671 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 9f25c338def749c39bd3bf23a3aa5088 (DEFAULT_TABLE table=test-table [id=dd20058392584d6e9885892207c072e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:47.561781 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f25c338def749c39bd3bf23a3aa5088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:47.561795 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f25c338def749c39bd3bf23a3aa5088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:47.562795 2704 tablet_bootstrap.cc:492] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: Bootstrap starting.
I20260927 02:52:47.562716 2703 tablet_bootstrap.cc:492] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: Bootstrap starting.
I20260927 02:52:47.562832 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 9f25c338def749c39bd3bf23a3aa5088 (DEFAULT_TABLE table=test-table [id=dd20058392584d6e9885892207c072e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:47.562930 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f25c338def749c39bd3bf23a3aa5088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:47.563238 2704 tablet_bootstrap.cc:654] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:47.563247 2703 tablet_bootstrap.cc:654] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:47.563755 2703 tablet_bootstrap.cc:492] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: No bootstrap required, opened a new log
I20260927 02:52:47.563793 2703 ts_tablet_manager.cc:1397] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:47.563910 2703 raft_consensus.cc:348] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.563969 2703 raft_consensus.cc:374] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:47.563988 2703 raft_consensus.cc:729] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07866aa5f3c48c7883e9bcc3d300f48, State: Initialized, Role: FOLLOWER
I20260927 02:52:47.564035 2703 consensus_queue.cc:260] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.564162 2707 tablet_bootstrap.cc:492] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: Bootstrap starting.
I20260927 02:52:47.564172 2703 ts_tablet_manager.cc:1428] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:47.564442 2704 tablet_bootstrap.cc:492] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: No bootstrap required, opened a new log
I20260927 02:52:47.564476 2707 tablet_bootstrap.cc:654] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:47.564491 2704 ts_tablet_manager.cc:1397] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:52:47.564618 2704 raft_consensus.cc:348] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.564692 2704 raft_consensus.cc:374] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:47.564711 2704 raft_consensus.cc:729] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Initialized, Role: FOLLOWER
I20260927 02:52:47.564764 2704 consensus_queue.cc:260] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.564908 2704 ts_tablet_manager.cc:1428] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:47.565201 2707 tablet_bootstrap.cc:492] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: No bootstrap required, opened a new log
I20260927 02:52:47.565246 2707 ts_tablet_manager.cc:1397] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:47.565397 2707 raft_consensus.cc:348] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.565454 2707 raft_consensus.cc:374] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:47.565472 2707 raft_consensus.cc:729] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eded432842a44073afcaf37cbcfdbb86, State: Initialized, Role: FOLLOWER
I20260927 02:52:47.565510 2707 consensus_queue.cc:260] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.565641 2707 ts_tablet_manager.cc:1428] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:47.622864 2631 raft_consensus.cc:482] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:47.622954 2631 raft_consensus.cc:504] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.623077 2631 leader_election.cc:288] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:47.623293 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f25c338def749c39bd3bf23a3aa5088" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86" is_pre_election: true
I20260927 02:52:47.623273 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f25c338def749c39bd3bf23a3aa5088" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" is_pre_election: true
I20260927 02:52:47.623373 2414 raft_consensus.cc:2454] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:47.623373 2553 raft_consensus.cc:2454] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:47.623508 2205 leader_election.cc:302] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, f07866aa5f3c48c7883e9bcc3d300f48; no voters:
I20260927 02:52:47.623610 2631 raft_consensus.cc:2781] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:47.623661 2631 raft_consensus.cc:482] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:47.623680 2631 raft_consensus.cc:3037] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:47.624336 2631 raft_consensus.cc:504] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.624459 2631 leader_election.cc:288] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Requested vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:47.624579 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f25c338def749c39bd3bf23a3aa5088" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48"
I20260927 02:52:47.624604 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f25c338def749c39bd3bf23a3aa5088" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86"
I20260927 02:52:47.624648 2414 raft_consensus.cc:3037] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:47.624660 2553 raft_consensus.cc:3037] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:47.625188 2414 raft_consensus.cc:2454] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:47.625349 2205 leader_election.cc:302] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, f07866aa5f3c48c7883e9bcc3d300f48; no voters:
I20260927 02:52:47.625448 2631 raft_consensus.cc:2781] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:47.625505 2631 raft_consensus.cc:686] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Running, Role: LEADER
I20260927 02:52:47.625560 2631 consensus_queue.cc:237] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.625698 2553 raft_consensus.cc:2454] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:47.626151 2116 catalog_manager.cc:5495] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } health_report { overall_health: UNKNOWN } } }
I20260927 02:52:47.638183 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:47.636301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.0.98.1:44181)
0927 02:52:47.636427 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:47.636429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:47.636437 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:47.636457 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:47.636458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:47.636469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:47.636523 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:47.636617 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:47.637079 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:47.637729 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:47.637917 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:47.637957 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:47.638039 (+ 82us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:47.638061 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:47.638090 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:47.638120 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 02:52:47.638133 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":93,"threads_started":1}
I20260927 02:52:47.639954 2553 raft_consensus.cc:1261] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:47.639967 2414 raft_consensus.cc:1261] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:47.640133 2675 consensus_queue.cc:1035] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:47.640236 2680 consensus_queue.cc:1035] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:47.648504 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:47.648578 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:47.649081 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: Sending DeleteTablet for 3 replicas of tablet 9f25c338def749c39bd3bf23a3aa5088
I20260927 02:52:47.649255 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f25c338def749c39bd3bf23a3aa5088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54524
I20260927 02:52:47.649307 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f25c338def749c39bd3bf23a3aa5088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53686
I20260927 02:52:47.649338 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f25c338def749c39bd3bf23a3aa5088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34706
I20260927 02:52:47.649410 2694 tablet_replica.cc:330] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: stopping tablet replica
I20260927 02:52:47.649441 2692 tablet_replica.cc:330] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: stopping tablet replica
I20260927 02:52:47.649461 2694 raft_consensus.cc:2229] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:47.649495 2692 raft_consensus.cc:2229] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:52:47.649510 2694 raft_consensus.cc:2258] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:47.649545 2692 raft_consensus.cc:2258] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:47.649616 2693 tablet_replica.cc:330] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: stopping tablet replica
I20260927 02:52:47.649663 2693 raft_consensus.cc:2229] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:47.649705 2693 raft_consensus.cc:2258] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:47.649735 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:52:47.649824 2694 ts_tablet_manager.cc:1905] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:52:47.649855 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:47.649907 2692 ts_tablet_manager.cc:1905] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:47.651041 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2f22db516545e7bbe82cc335105e67 (DEFAULT_TABLE table=test-table [id=6543e14286d946eab971b06dd5af5e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:47.651084 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2f22db516545e7bbe82cc335105e67 (DEFAULT_TABLE table=test-table [id=6543e14286d946eab971b06dd5af5e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:47.651135 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2f22db516545e7bbe82cc335105e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:47.651166 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2f22db516545e7bbe82cc335105e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:47.651412 2692 ts_tablet_manager.cc:1918] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:47.651463 2692 log.cc:1196] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/9f25c338def749c39bd3bf23a3aa5088
I20260927 02:52:47.651631 2692 ts_tablet_manager.cc:1939] T 9f25c338def749c39bd3bf23a3aa5088 P 6dbccc61d945442a9cccc5c6950ece27: Deleting consensus metadata
I20260927 02:52:47.651916 2693 ts_tablet_manager.cc:1905] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:47.652289 2704 tablet_bootstrap.cc:492] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27: Bootstrap starting.
I20260927 02:52:47.652565 2704 tablet_bootstrap.cc:654] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:47.652578 2694 ts_tablet_manager.cc:1918] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:47.652630 2694 log.cc:1196] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/9f25c338def749c39bd3bf23a3aa5088
I20260927 02:52:47.652809 2694 ts_tablet_manager.cc:1939] T 9f25c338def749c39bd3bf23a3aa5088 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting consensus metadata
I20260927 02:52:47.653056 2704 tablet_bootstrap.cc:492] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27: No bootstrap required, opened a new log
I20260927 02:52:47.653084 2101 catalog_manager.cc:4841] TS f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427): tablet 9f25c338def749c39bd3bf23a3aa5088 (table test-table [id=dd20058392584d6e9885892207c072e4]) successfully deleted
I20260927 02:52:47.653100 2704 ts_tablet_manager.cc:1397] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:47.653126 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2f22db516545e7bbe82cc335105e67 (DEFAULT_TABLE table=test-table [id=6543e14286d946eab971b06dd5af5e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:47.653163 2693 ts_tablet_manager.cc:1918] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:52:47.653201 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2f22db516545e7bbe82cc335105e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:47.653201 2693 log.cc:1196] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/9f25c338def749c39bd3bf23a3aa5088
I20260927 02:52:47.653203 2704 raft_consensus.cc:348] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.653259 2704 raft_consensus.cc:374] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:47.653277 2704 raft_consensus.cc:729] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Initialized, Role: FOLLOWER
I20260927 02:52:47.653316 2704 consensus_queue.cc:260] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.653381 2693 ts_tablet_manager.cc:1939] T 9f25c338def749c39bd3bf23a3aa5088 P eded432842a44073afcaf37cbcfdbb86: Deleting consensus metadata
I20260927 02:52:47.653406 2704 ts_tablet_manager.cc:1428] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:47.653482 2707 tablet_bootstrap.cc:492] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: Bootstrap starting.
I20260927 02:52:47.653843 2707 tablet_bootstrap.cc:654] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:47.654016 2703 tablet_bootstrap.cc:492] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: Bootstrap starting.
I20260927 02:52:47.654170 2104 catalog_manager.cc:4841] TS eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119): tablet 9f25c338def749c39bd3bf23a3aa5088 (table test-table [id=dd20058392584d6e9885892207c072e4]) successfully deleted
I20260927 02:52:47.654444 2101 catalog_manager.cc:4841] TS 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1:44181): tablet 9f25c338def749c39bd3bf23a3aa5088 (table test-table [id=dd20058392584d6e9885892207c072e4]) successfully deleted
I20260927 02:52:47.654495 2707 tablet_bootstrap.cc:492] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: No bootstrap required, opened a new log
I20260927 02:52:47.654538 2707 ts_tablet_manager.cc:1397] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:47.654573 2703 tablet_bootstrap.cc:654] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:47.654680 2707 raft_consensus.cc:348] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.654742 2707 raft_consensus.cc:374] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:47.654778 2707 raft_consensus.cc:729] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eded432842a44073afcaf37cbcfdbb86, State: Initialized, Role: FOLLOWER
I20260927 02:52:47.654820 2707 consensus_queue.cc:260] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.654904 2707 ts_tablet_manager.cc:1428] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:47.655198 2703 tablet_bootstrap.cc:492] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: No bootstrap required, opened a new log
I20260927 02:52:47.655246 2703 ts_tablet_manager.cc:1397] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:47.655369 2703 raft_consensus.cc:348] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.655421 2703 raft_consensus.cc:374] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:47.655433 2703 raft_consensus.cc:729] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07866aa5f3c48c7883e9bcc3d300f48, State: Initialized, Role: FOLLOWER
I20260927 02:52:47.655465 2703 consensus_queue.cc:260] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.655541 2703 ts_tablet_manager.cc:1428] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:47.918111 2680 raft_consensus.cc:482] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:47.918210 2680 raft_consensus.cc:504] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.918365 2680 leader_election.cc:288] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:47.918578 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2f22db516545e7bbe82cc335105e67" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86" is_pre_election: true
I20260927 02:52:47.918550 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2f22db516545e7bbe82cc335105e67" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" is_pre_election: true
I20260927 02:52:47.918665 2553 raft_consensus.cc:2454] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:47.918665 2414 raft_consensus.cc:2454] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 0.
I20260927 02:52:47.918814 2208 leader_election.cc:302] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, eded432842a44073afcaf37cbcfdbb86; no voters:
I20260927 02:52:47.918916 2680 raft_consensus.cc:2781] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:47.918949 2680 raft_consensus.cc:482] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:47.918975 2680 raft_consensus.cc:3037] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:47.919647 2680 raft_consensus.cc:504] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.919761 2680 leader_election.cc:288] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Requested vote from peers f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427), eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119)
I20260927 02:52:47.919899 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2f22db516545e7bbe82cc335105e67" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07866aa5f3c48c7883e9bcc3d300f48"
I20260927 02:52:47.919968 2414 raft_consensus.cc:3037] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:47.919942 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2f22db516545e7bbe82cc335105e67" candidate_uuid: "6dbccc61d945442a9cccc5c6950ece27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded432842a44073afcaf37cbcfdbb86"
I20260927 02:52:47.920032 2553 raft_consensus.cc:3037] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:47.920703 2414 raft_consensus.cc:2454] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:47.920737 2553 raft_consensus.cc:2454] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dbccc61d945442a9cccc5c6950ece27 in term 1.
I20260927 02:52:47.920871 2205 leader_election.cc:302] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dbccc61d945442a9cccc5c6950ece27, f07866aa5f3c48c7883e9bcc3d300f48; no voters:
I20260927 02:52:47.920965 2680 raft_consensus.cc:2781] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:47.921015 2680 raft_consensus.cc:686] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbccc61d945442a9cccc5c6950ece27, State: Running, Role: LEADER
I20260927 02:52:47.921061 2680 consensus_queue.cc:237] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } }
I20260927 02:52:47.921576 2115 catalog_manager.cc:5495] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6dbccc61d945442a9cccc5c6950ece27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dbccc61d945442a9cccc5c6950ece27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } health_report { overall_health: UNKNOWN } } }
I20260927 02:52:47.935986 2414 raft_consensus.cc:1261] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:47.935987 2553 raft_consensus.cc:1261] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Refusing update from remote peer 6dbccc61d945442a9cccc5c6950ece27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:47.936188 2631 consensus_queue.cc:1035] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f07866aa5f3c48c7883e9bcc3d300f48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:47.936291 2680 consensus_queue.cc:1035] T 4b2f22db516545e7bbe82cc335105e67 P 6dbccc61d945442a9cccc5c6950ece27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eded432842a44073afcaf37cbcfdbb86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:52:47.939774 2673 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:52:49.454416 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:49.454505 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:49.455211 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8494b0354d0444f983d9b72b847100fd: Sending DeleteTablet for 3 replicas of tablet 4b2f22db516545e7bbe82cc335105e67
I20260927 02:52:49.455449 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2f22db516545e7bbe82cc335105e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53686
I20260927 02:52:49.455458 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2f22db516545e7bbe82cc335105e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54524
I20260927 02:52:49.455586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 2189
I20260927 02:52:49.455636 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2f22db516545e7bbe82cc335105e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34706
I20260927 02:52:49.456610 2745 tablet_replica.cc:330] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: stopping tablet replica
I20260927 02:52:49.456676 2745 raft_consensus.cc:2229] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:49.456712 2745 raft_consensus.cc:2258] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:49.456934 2745 ts_tablet_manager.cc:1905] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:49.458539 2743 tablet_replica.cc:330] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: stopping tablet replica
I20260927 02:52:49.458611 2743 raft_consensus.cc:2229] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:49.458645 2743 raft_consensus.cc:2258] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:49.458827 2745 ts_tablet_manager.cc:1918] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:52:49.458890 2743 ts_tablet_manager.cc:1905] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:49.458920 2745 log.cc:1196] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/4b2f22db516545e7bbe82cc335105e67
I20260927 02:52:49.459116 2745 ts_tablet_manager.cc:1939] T 4b2f22db516545e7bbe82cc335105e67 P eded432842a44073afcaf37cbcfdbb86: Deleting consensus metadata
I20260927 02:52:49.459429 2104 catalog_manager.cc:4841] TS eded432842a44073afcaf37cbcfdbb86 (127.0.98.3:41119): tablet 4b2f22db516545e7bbe82cc335105e67 (table test-table [id=6543e14286d946eab971b06dd5af5e2b]) successfully deleted
I20260927 02:52:49.460254 2743 ts_tablet_manager.cc:1918] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:52:49.460312 2743 log.cc:1196] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/4b2f22db516545e7bbe82cc335105e67
I20260927 02:52:49.460527 2743 ts_tablet_manager.cc:1939] T 4b2f22db516545e7bbe82cc335105e67 P f07866aa5f3c48c7883e9bcc3d300f48: Deleting consensus metadata
I20260927 02:52:49.460871 2101 catalog_manager.cc:4841] TS f07866aa5f3c48c7883e9bcc3d300f48 (127.0.98.2:45427): tablet 4b2f22db516545e7bbe82cc335105e67 (table test-table [id=6543e14286d946eab971b06dd5af5e2b]) successfully deleted
W20260927 02:52:49.461962 2101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44181 (error 108)
I20260927 02:52:49.462198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 2329
I20260927 02:52:49.462819 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:49.462282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44181 (local address 127.0.0.1:41622)
0927 02:52:49.462713 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:49.462780 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":51,"threads_started":1}
W20260927 02:52:49.462985 2101 catalog_manager.cc:4584] TS 6dbccc61d945442a9cccc5c6950ece27 (127.0.98.1:44181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b2f22db516545e7bbe82cc335105e67: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44181: connect: Connection refused (error 111)
I20260927 02:52:49.466830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 2469
I20260927 02:52:49.470834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 2085
2026-09-27T02:52:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5803 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:49 dist-test-slave-mfx9 krb5kdc[2750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:49 dist-test-slave-mfx9 krb5kdc[2750](info): set up 2 sockets
Sep 27 02:52:49 dist-test-slave-mfx9 krb5kdc[2750](info): commencing operation
krb5kdc: starting...
W20260927 02:52:50.418234 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:52:50 dist-test-slave-mfx9 krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:52:50.525358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:50.604422 2763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:50.604612 2763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:50.604631 2763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:50.605715 2763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:50.605748 2763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:50.605760 2763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:50.605777 2763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:50.605794 2763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:50.607249 2763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:50.607431 2763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:50.607592 2763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:50.611986 2769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:50.611984 2771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:50.612062 2763 server_base.cc:1034] running on GCE node
W20260927 02:52:50.611964 2768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:50.612423 2763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:50.612658 2763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:50.613787 2763 hybrid_clock.cc:648] HybridClock initialized: now 1790477570613779 us; error 27 us; skew 500 ppm
Sep 27 02:52:50 dist-test-slave-mfx9 krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:50.619318 2763 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:50.619743 2763 webserver.cc:460] Webserver started at http://127.0.98.62:45481/ using document root <none> and password file <none>
I20260927 02:52:50.619921 2763 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:50.619959 2763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:50.620048 2763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:50.621057 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "4447cf6b98b24a2d9faac83b00af8b92"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "a1262aa2a333b9a95cc8cbef4ecc8ac5"
server_key_iv: "78cadef4c1984656063f91f4e0fb0f8c"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.621384 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "4447cf6b98b24a2d9faac83b00af8b92"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "a1262aa2a333b9a95cc8cbef4ecc8ac5"
server_key_iv: "78cadef4c1984656063f91f4e0fb0f8c"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.622572 2763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:52:50.623253 2778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:50.623386 2763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:50.623461 2763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "4447cf6b98b24a2d9faac83b00af8b92"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "a1262aa2a333b9a95cc8cbef4ecc8ac5"
server_key_iv: "78cadef4c1984656063f91f4e0fb0f8c"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.623522 2763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:50.630573 2763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:50.632174 2763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:50.632283 2763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:50.635854 2763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44231
I20260927 02:52:50.635895 2830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44231 every 8 connection(s)
I20260927 02:52:50.636304 2763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:50.636868 2831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:50.638983 2831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92: Bootstrap starting.
I20260927 02:52:50.639537 2831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:50.639519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 2763
I20260927 02:52:50.639607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:50.639789 392 external_mini_cluster.cc:1428] Setting key 8b0c00888919938376e2e1c564e6a0ef
I20260927 02:52:50.639837 2831 log.cc:824] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:50.640559 2831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92: No bootstrap required, opened a new log
I20260927 02:52:50.642970 2831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4447cf6b98b24a2d9faac83b00af8b92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44231 } }
I20260927 02:52:50.643123 2831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:50.643172 2831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4447cf6b98b24a2d9faac83b00af8b92, State: Initialized, Role: FOLLOWER
I20260927 02:52:50.643290 2831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4447cf6b98b24a2d9faac83b00af8b92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44231 } }
I20260927 02:52:50.643350 2831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:50.643369 2831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:50.643419 2831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:50.643889 2831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4447cf6b98b24a2d9faac83b00af8b92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44231 } }
I20260927 02:52:50.643985 2831 leader_election.cc:302] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4447cf6b98b24a2d9faac83b00af8b92; no voters:
I20260927 02:52:50.644115 2831 leader_election.cc:288] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:50.644152 2836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:50.644279 2836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [term 1 LEADER]: Becoming Leader. State: Replica: 4447cf6b98b24a2d9faac83b00af8b92, State: Running, Role: LEADER
I20260927 02:52:50.644330 2831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:50.644407 2836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4447cf6b98b24a2d9faac83b00af8b92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44231 } }
I20260927 02:52:50.644701 2836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4447cf6b98b24a2d9faac83b00af8b92. Latest consensus state: current_term: 1 leader_uuid: "4447cf6b98b24a2d9faac83b00af8b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4447cf6b98b24a2d9faac83b00af8b92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44231 } } }
I20260927 02:52:50.644749 2838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4447cf6b98b24a2d9faac83b00af8b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4447cf6b98b24a2d9faac83b00af8b92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44231 } } }
I20260927 02:52:50.644845 2838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:50.645073 2836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:50.645185 2845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:50.645689 2845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:50.647199 2845 catalog_manager.cc:1347] Generated new cluster ID: 09a651931570406ca684c144e003a31a
I20260927 02:52:50.647250 2845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:52:50 dist-test-slave-mfx9 krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:50.651276 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:50.640280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.0.98.62:44231)
0927 02:52:50.640452 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:50.640461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:50.640492 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:52:50.640976 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:50.640985 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:50.641043 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:50.641184 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:50.641698 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.643044 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.643874 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.644458 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.644835 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:50.648338 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:50.648342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:50.648342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:50.648353 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:50.649794 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:50.650097 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:50.650100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:50.650101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:50.650176 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:50.650341 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:50.650341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:50.650342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:50.650575 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:50.650710 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:50.650828 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 02:52:50.650950 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260927 02:52:50.657114 2845 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:50.657639 2845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:50.669678 2845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4447cf6b98b24a2d9faac83b00af8b92: Generated new TSK 0
I20260927 02:52:50.669869 2845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:52:50.697875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44231
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:50.780614 2857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:50.780809 2857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:50.780831 2857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:50.781967 2857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:50.782014 2857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:50.782040 2857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:50.783577 2857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:50.783768 2857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:50.783943 2857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:50.788667 2863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:50.788758 2857 server_base.cc:1034] running on GCE node
W20260927 02:52:50.788775 2865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:50.788661 2862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:50.789146 2857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:50.789363 2857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:50.790494 2857 hybrid_clock.cc:648] HybridClock initialized: now 1790477570790483 us; error 27 us; skew 500 ppm
Sep 27 02:52:50 dist-test-slave-mfx9 krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:50.792867 2857 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:52:50.793332 2857 webserver.cc:460] Webserver started at http://127.0.98.1:38739/ using document root <none> and password file <none>
I20260927 02:52:50.793562 2857 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:50.793609 2857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:50.793728 2857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:50.794724 2857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "cfbaea37117849d8b6020af6697b87fd"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "e8794e28b94ccd11bc8ba4a0ba644270"
server_key_iv: "bc8ab00089d655a051841587dfa7fe30"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.795097 2857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cfbaea37117849d8b6020af6697b87fd"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "e8794e28b94ccd11bc8ba4a0ba644270"
server_key_iv: "bc8ab00089d655a051841587dfa7fe30"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.796274 2857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:50.796922 2872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:50.797067 2857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:50.797143 2857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "cfbaea37117849d8b6020af6697b87fd"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "e8794e28b94ccd11bc8ba4a0ba644270"
server_key_iv: "bc8ab00089d655a051841587dfa7fe30"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.797207 2857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:50.820344 2857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:50.822017 2857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:50.822144 2857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:50.822335 2857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:50.822609 2857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:50.822641 2857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:50.822676 2857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:50.822700 2857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:50.828044 2857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44447
I20260927 02:52:50.828080 2985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44447 every 8 connection(s)
I20260927 02:52:50.828352 2857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:52:50.832680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 2857
I20260927 02:52:50.832767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:50.832942 392 external_mini_cluster.cc:1428] Setting key c25364029366e73b96a18e8a904e685a
Sep 27 02:52:50 dist-test-slave-mfx9 krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:50.837049 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:50.828989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49127 (local address 127.0.98.62:44231)
0927 02:52:50.829034 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:50.829037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:50.829222 (+ 185us) server_negotiation.cc:407] Connection header received
0927 02:52:50.830078 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:50.830079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:50.830088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:50.830137 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:50.830665 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.831201 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.832110 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.832300 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.832347 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:50.834567 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:50.834569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:50.834569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:50.834576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:50.835827 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:50.836235 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:50.836235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:50.836236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:50.836279 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:50.836478 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:50.836479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:50.836479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:50.836566 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:50.836607 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:50.836918 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 02:52:50.836985 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:52:50.837265 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:50.828976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44231 (local address 127.0.98.1:49127)
0927 02:52:50.829113 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:50.829146 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:50.829999 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:50.830191 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:50.830194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:50.830426 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:50.830600 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:50.830604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.831263 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.831263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:50.832049 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:50.832051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.832351 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.832352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:50.832451 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:50.832918 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:50.832928 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:50.834467 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:50.835907 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:50.835913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:50.835915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:50.836172 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:50.836316 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:50.836316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:50.836317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:50.836429 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:50.836617 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:50.836622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:50.836779 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 02:52:50.836900 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 02:52:50.837052 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260927 02:52:50.837710 2986 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44231
I20260927 02:52:50.837800 2986 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:50.837991 2986 heartbeater.cc:511] Master 127.0.98.62:44231 requested a full tablet report, sending...
I20260927 02:52:50.838482 2795 ts_manager.cc:194] Registered new tserver with Master: cfbaea37117849d8b6020af6697b87fd (127.0.98.1:44447)
I20260927 02:52:50.839145 2795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49127
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:52:50.855350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44231
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:50.933554 2991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:50.933753 2991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:50.933773 2991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:50.934907 2991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:50.934947 2991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:50.934973 2991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:50.936434 2991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:50.936641 2991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:50.936838 2991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:50.941310 2996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:50.941310 2997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:50.941306 2999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:50.941462 2991 server_base.cc:1034] running on GCE node
I20260927 02:52:50.941720 2991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:50.941934 2991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:50.943058 2991 hybrid_clock.cc:648] HybridClock initialized: now 1790477570943039 us; error 32 us; skew 500 ppm
Sep 27 02:52:50 dist-test-slave-mfx9 krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:50.945269 2991 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:52:50.945638 2991 webserver.cc:460] Webserver started at http://127.0.98.2:43599/ using document root <none> and password file <none>
I20260927 02:52:50.945823 2991 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:50.945863 2991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:50.945977 2991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:50.946982 2991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "21228d48470c463fa365f3b66e3c53ce"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "07afa26e4c8fe259d112e00f87da07c2"
server_key_iv: "baac8b1d94a855444390be6be11894a0"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.947260 2991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21228d48470c463fa365f3b66e3c53ce"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "07afa26e4c8fe259d112e00f87da07c2"
server_key_iv: "baac8b1d94a855444390be6be11894a0"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.948333 2991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:50.948943 3006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:50.949081 2991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:50.949152 2991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "21228d48470c463fa365f3b66e3c53ce"
format_stamp: "Formatted at 2026-09-27 02:52:50 on dist-test-slave-mfx9"
server_key: "07afa26e4c8fe259d112e00f87da07c2"
server_key_iv: "baac8b1d94a855444390be6be11894a0"
server_key_version: "encryptionkey@0"
I20260927 02:52:50.949219 2991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:50.969987 2991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:50.971664 2991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:50.971767 2991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:50.971931 2991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:50.972164 2991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:50.972193 2991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:50.972211 2991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:50.972227 2991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:50.977417 2991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42581
I20260927 02:52:50.977456 3119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42581 every 8 connection(s)
I20260927 02:52:50.977748 2991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:52:50.979131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 2991
I20260927 02:52:50.979212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:50.979362 392 external_mini_cluster.cc:1428] Setting key 2d85884466a5c873fb38ca25adf02de8
Sep 27 02:52:50 dist-test-slave-mfx9 krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:52:50.987725 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:50.978429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32921 (local address 127.0.98.62:44231)
0927 02:52:50.978506 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:50.978508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:50.978632 (+ 124us) server_negotiation.cc:407] Connection header received
0927 02:52:50.979613 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:50.979614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:50.979624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:50.979697 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:50.980464 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.981130 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.982495 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.982675 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.982723 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:50.985067 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:50.985068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:50.985069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:50.985075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:50.986404 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:50.986872 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:50.986873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:50.986873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:50.986920 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:50.987157 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:50.987157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:50.987157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:50.987248 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:50.987301 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:50.987602 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 02:52:50.987661 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:52:50.987897 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:50.978346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44231 (local address 127.0.98.2:32921)
0927 02:52:50.978494 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:50.978534 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:50.979515 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:50.979817 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:50.979827 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:50.980181 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:50.980369 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:50.980373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.981215 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.981216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:50.982420 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:50.982423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:50.982801 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:50.982802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:50.982932 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:50.983429 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:50.983439 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:50.984979 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:50.986479 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:50.986486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:50.986488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:50.986798 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:50.986975 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:50.986976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:50.986977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:50.987106 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:50.987303 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:50.987308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:50.987469 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 02:52:50.987573 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 02:52:50.987699 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260927 02:52:50.988325 3120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44231
I20260927 02:52:50.988421 3120 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:50.988590 3120 heartbeater.cc:511] Master 127.0.98.62:44231 requested a full tablet report, sending...
I20260927 02:52:50.988914 2795 ts_manager.cc:194] Registered new tserver with Master: 21228d48470c463fa365f3b66e3c53ce (127.0.98.2:42581)
I20260927 02:52:50.989348 2795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:52:51.002432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44231
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:51.079242 3125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:51.079449 3125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:51.079471 3125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:51.080637 3125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:51.080681 3125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:51.080709 3125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:51.082178 3125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:51.082398 3125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:51.082607 3125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:51.087301 3133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:51.087313 3131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:51.087334 3125 server_base.cc:1034] running on GCE node
W20260927 02:52:51.087334 3130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:51.087738 3125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:51.087958 3125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:51.089099 3125 hybrid_clock.cc:648] HybridClock initialized: now 1790477571089080 us; error 32 us; skew 500 ppm
Sep 27 02:52:51 dist-test-slave-mfx9 krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:51.091511 3125 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:52:51.091902 3125 webserver.cc:460] Webserver started at http://127.0.98.3:35177/ using document root <none> and password file <none>
I20260927 02:52:51.092095 3125 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:51.092140 3125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:51.092262 3125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:51.093246 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "e512602282d64ade9fa0e52fbd925637"
format_stamp: "Formatted at 2026-09-27 02:52:51 on dist-test-slave-mfx9"
server_key: "7c8cf2826f446ad89b2512a582476a67"
server_key_iv: "df919af5583db75217cf44ca2d4fde56"
server_key_version: "encryptionkey@0"
I20260927 02:52:51.093564 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e512602282d64ade9fa0e52fbd925637"
format_stamp: "Formatted at 2026-09-27 02:52:51 on dist-test-slave-mfx9"
server_key: "7c8cf2826f446ad89b2512a582476a67"
server_key_iv: "df919af5583db75217cf44ca2d4fde56"
server_key_version: "encryptionkey@0"
I20260927 02:52:51.094733 3125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:51.095377 3140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:51.095516 3125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:51.095592 3125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "e512602282d64ade9fa0e52fbd925637"
format_stamp: "Formatted at 2026-09-27 02:52:51 on dist-test-slave-mfx9"
server_key: "7c8cf2826f446ad89b2512a582476a67"
server_key_iv: "df919af5583db75217cf44ca2d4fde56"
server_key_version: "encryptionkey@0"
I20260927 02:52:51.095649 3125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:51.113718 3125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:51.115527 3125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:51.115630 3125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:51.115796 3125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:51.116045 3125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:51.116073 3125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:51.116093 3125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:51.116108 3125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:51.121276 3125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45215
I20260927 02:52:51.121351 3253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45215 every 8 connection(s)
I20260927 02:52:51.121613 3125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:52:51.126292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 3125
I20260927 02:52:51.126384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:51.126545 392 external_mini_cluster.cc:1428] Setting key 56a6d8a8456e40f2b10f388fa86d404d
Sep 27 02:52:51 dist-test-slave-mfx9 krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:51.130038 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:51.122244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51129 (local address 127.0.98.62:44231)
0927 02:52:51.122286 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:51.122288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:51.122465 (+ 177us) server_negotiation.cc:407] Connection header received
0927 02:52:51.123381 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:51.123382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:51.123402 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:51.123451 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:51.124039 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:51.124533 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:51.125512 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:51.125721 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:51.125776 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:51.127852 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:51.127853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:51.127854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:51.127861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:51.128903 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:51.129232 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:51.129233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:51.129234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:51.129283 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:51.129483 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:51.129484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:51.129484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:51.129587 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:51.129639 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:51.129878 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:52:51.129953 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:52:51.130206 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:51.122209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44231 (local address 127.0.98.3:51129)
0927 02:52:51.122343 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:51.122382 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:51.123280 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:51.123520 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:51.123524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:51.123770 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:51.123957 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:51.123961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:51.124608 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:51.124609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:51.125416 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:51.125418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:51.125789 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:51.125790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:51.125916 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:51.126350 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:51.126359 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:51.127772 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:51.128970 (+ 1198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:51.128975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:51.128977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:51.129172 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:51.129333 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:51.129333 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:51.129333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:51.129432 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:51.129637 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:51.129643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:51.129770 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 02:52:51.129853 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:52:51.130007 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260927 02:52:51.130581 3254 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44231
I20260927 02:52:51.130658 3254 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:51.130856 3254 heartbeater.cc:511] Master 127.0.98.62:44231 requested a full tablet report, sending...
I20260927 02:52:51.131174 2795 ts_manager.cc:194] Registered new tserver with Master: e512602282d64ade9fa0e52fbd925637 (127.0.98.3:45215)
I20260927 02:52:51.131505 2795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51129
I20260927 02:52:51.138187 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:52:51.147753 2835 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:52:51.146992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.0.98.62:44231)
0927 02:52:51.147031 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:51.147033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:51.147122 (+ 89us) server_negotiation.cc:407] Connection header received
0927 02:52:51.147173 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:51.147174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:51.147181 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:51.147248 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:51.147691 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54638: BlockingRecv error: recv got EOF from 127.0.0.1:54638 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:52:51.148648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 2857
I20260927 02:52:51.152447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 2991
I20260927 02:52:51.156261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 3125
I20260927 02:52:51.159809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 2763
2026-09-27T02:52:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1689 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:51 dist-test-slave-mfx9 krb5kdc[3266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:51 dist-test-slave-mfx9 krb5kdc[3266](info): set up 2 sockets
Sep 27 02:52:51 dist-test-slave-mfx9 krb5kdc[3266](info): commencing operation
krb5kdc: starting...
W20260927 02:52:52.102319 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:52:52 dist-test-slave-mfx9 krb5kdc[3266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:52:52.209313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:52.286219 3279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:52.286414 3279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:52.286442 3279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:52.287580 3279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:52.287619 3279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:52.287637 3279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:52.287654 3279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:52.287668 3279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:52.289160 3279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:52.289391 3279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:52.289579 3279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:52.293999 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:52.294011 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:52.294044 3279 server_base.cc:1034] running on GCE node
W20260927 02:52:52.294068 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:52.294366 3279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:52.294620 3279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:52.295751 3279 hybrid_clock.cc:648] HybridClock initialized: now 1790477572295728 us; error 39 us; skew 500 ppm
Sep 27 02:52:52 dist-test-slave-mfx9 krb5kdc[3266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:52.297974 3279 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:52.298347 3279 webserver.cc:460] Webserver started at http://127.0.98.62:33903/ using document root <none> and password file <none>
I20260927 02:52:52.298545 3279 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:52.298591 3279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:52.298712 3279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:52.299721 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "b58ae401524a448fa5de33f247d9dce7"
format_stamp: "Formatted at 2026-09-27 02:52:52 on dist-test-slave-mfx9"
server_key: "73124197102b826bcd7d6a23c94f2a7d"
server_key_iv: "fb1062d92ad1389f72887089b7aa3eb9"
server_key_version: "encryptionkey@0"
I20260927 02:52:52.300035 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "b58ae401524a448fa5de33f247d9dce7"
format_stamp: "Formatted at 2026-09-27 02:52:52 on dist-test-slave-mfx9"
server_key: "73124197102b826bcd7d6a23c94f2a7d"
server_key_iv: "fb1062d92ad1389f72887089b7aa3eb9"
server_key_version: "encryptionkey@0"
I20260927 02:52:52.301179 3279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:52:52.301755 3294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:52.301915 3279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:52.301990 3279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "b58ae401524a448fa5de33f247d9dce7"
format_stamp: "Formatted at 2026-09-27 02:52:52 on dist-test-slave-mfx9"
server_key: "73124197102b826bcd7d6a23c94f2a7d"
server_key_iv: "fb1062d92ad1389f72887089b7aa3eb9"
server_key_version: "encryptionkey@0"
I20260927 02:52:52.302048 3279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:52.312150 3279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:52.313655 3279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:52.313771 3279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:52.317180 3279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36681
I20260927 02:52:52.317224 3346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36681 every 8 connection(s)
I20260927 02:52:52.317502 3279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:52.318292 3347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:52.320371 3347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7: Bootstrap starting.
I20260927 02:52:52.320868 3347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:52.321081 3347 log.cc:824] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:52.321692 3347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7: No bootstrap required, opened a new log
I20260927 02:52:52.323125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 3279
I20260927 02:52:52.323199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:52.323335 392 external_mini_cluster.cc:1428] Setting key 59386bbd3a01a841e7574009e3650057
I20260927 02:52:52.323311 3347 raft_consensus.cc:348] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58ae401524a448fa5de33f247d9dce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } }
I20260927 02:52:52.323472 3347 raft_consensus.cc:374] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:52.323518 3347 raft_consensus.cc:729] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58ae401524a448fa5de33f247d9dce7, State: Initialized, Role: FOLLOWER
I20260927 02:52:52.323653 3347 consensus_queue.cc:260] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58ae401524a448fa5de33f247d9dce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } }
I20260927 02:52:52.323772 3347 raft_consensus.cc:388] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:52.323834 3347 raft_consensus.cc:482] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:52.323889 3347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:52.324446 3347 raft_consensus.cc:504] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58ae401524a448fa5de33f247d9dce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } }
I20260927 02:52:52.324548 3347 leader_election.cc:302] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b58ae401524a448fa5de33f247d9dce7; no voters:
I20260927 02:52:52.324693 3347 leader_election.cc:288] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:52.324729 3352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:52.324878 3352 raft_consensus.cc:686] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [term 1 LEADER]: Becoming Leader. State: Replica: b58ae401524a448fa5de33f247d9dce7, State: Running, Role: LEADER
I20260927 02:52:52.324913 3347 sys_catalog.cc:564] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:52.325012 3352 consensus_queue.cc:237] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58ae401524a448fa5de33f247d9dce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } }
W20260927 02:52:52.326625 3367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:52:52.326714 3367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:52:52.326929 3353 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b58ae401524a448fa5de33f247d9dce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58ae401524a448fa5de33f247d9dce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } } }
I20260927 02:52:52.327016 3353 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:52.327193 3369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:52.327243 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b58ae401524a448fa5de33f247d9dce7. Latest consensus state: current_term: 1 leader_uuid: "b58ae401524a448fa5de33f247d9dce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58ae401524a448fa5de33f247d9dce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } } }
I20260927 02:52:52.327366 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:52.327415 3369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:52.329743 3369 catalog_manager.cc:1347] Generated new cluster ID: 8d681b6cfd51434c82862f8893355f7d
I20260927 02:52:52.329797 3369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:52:52 dist-test-slave-mfx9 krb5kdc[3266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:52.333832 3351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:52.323768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.0.98.62:36681)
0927 02:52:52.324264 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:52.324273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:52.324304 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:52:52.324411 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:52.324418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:52.324473 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:52.324610 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:52.325300 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:52.326625 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:52.327627 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:52.328100 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:52.328198 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:52.330603 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:52.330607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:52.330607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:52.330617 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:52.332468 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:52.332790 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:52.332792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:52.332793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:52.332886 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:52.333053 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:52.333054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:52.333054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:52.333265 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:52.333405 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:52.333493 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 02:52:52.333696 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":398,"threads_started":1}
I20260927 02:52:52.334645 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:52:52.343751 3369 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:52.344242 3369 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:52.351353 3351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:52.347468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.0.98.62:36681)
0927 02:52:52.347500 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:52.347503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:52.347617 (+ 114us) server_negotiation.cc:407] Connection header received
0927 02:52:52.347664 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:52.347665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:52.347678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:52.347730 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:52.348101 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:52.348857 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:52.349841 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:52.350170 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:52.350228 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:52.350924 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:52.350926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:52.350926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:52.351122 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:52.351141 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:52.351148 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:52.351206 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:52.351258 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 02:52:52.351282 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 02:52:52.353127 3369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b58ae401524a448fa5de33f247d9dce7: Generated new TSK 0
I20260927 02:52:52.353278 3369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:52:52.357923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 3279
2026-09-27T02:52:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1195 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:52 dist-test-slave-mfx9 krb5kdc[3380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:52 dist-test-slave-mfx9 krb5kdc[3380](info): set up 2 sockets
Sep 27 02:52:52 dist-test-slave-mfx9 krb5kdc[3380](info): commencing operation
krb5kdc: starting...
W20260927 02:52:53.302546 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:52:53 dist-test-slave-mfx9 krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:52:53.409101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38273
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:53.485472 3393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:53.485677 3393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:53.485697 3393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:53.486819 3393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:53.486858 3393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:53.486871 3393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:53.486881 3393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:53.486892 3393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:53.488291 3393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:53.488489 3393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:53.488663 3393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:53.493168 3399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:53.493260 3398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:53.493356 3393 server_base.cc:1034] running on GCE node
W20260927 02:52:53.493257 3401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:53.493659 3393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:53.493942 3393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:53.495100 3393 hybrid_clock.cc:648] HybridClock initialized: now 1790477573495079 us; error 30 us; skew 500 ppm
Sep 27 02:52:53 dist-test-slave-mfx9 krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:53.497421 3393 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:53.497851 3393 webserver.cc:460] Webserver started at http://127.0.98.62:45473/ using document root <none> and password file <none>
I20260927 02:52:53.498042 3393 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:53.498088 3393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:53.498209 3393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:53.499259 3393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "576865c21b004fe7ae49682a0b5b4415"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "ea5bb6673f8106471fcb8c06b4494c29"
server_key_iv: "354a5361037c553040a39741cc59ce55"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.499575 3393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "576865c21b004fe7ae49682a0b5b4415"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "ea5bb6673f8106471fcb8c06b4494c29"
server_key_iv: "354a5361037c553040a39741cc59ce55"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.500738 3393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:52:53.501379 3408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:53.501523 3393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:53.501600 3393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "576865c21b004fe7ae49682a0b5b4415"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "ea5bb6673f8106471fcb8c06b4494c29"
server_key_iv: "354a5361037c553040a39741cc59ce55"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.501654 3393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:53.513517 3393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:53.515511 3393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:53.515640 3393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:53.519282 3393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38273
I20260927 02:52:53.519328 3460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38273 every 8 connection(s)
I20260927 02:52:53.519632 3393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:53.520511 3461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:53.522575 3461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415: Bootstrap starting.
I20260927 02:52:53.522929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 3393
I20260927 02:52:53.523032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:53.523213 392 external_mini_cluster.cc:1428] Setting key c0719c4d15ab2c6d35e1a62c9e636603
I20260927 02:52:53.523236 3461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:53.523499 3461 log.cc:824] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:53.524201 3461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415: No bootstrap required, opened a new log
I20260927 02:52:53.526438 3461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576865c21b004fe7ae49682a0b5b4415" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } }
I20260927 02:52:53.526599 3461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:53.526647 3461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576865c21b004fe7ae49682a0b5b4415, State: Initialized, Role: FOLLOWER
I20260927 02:52:53.526786 3461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576865c21b004fe7ae49682a0b5b4415" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } }
I20260927 02:52:53.526881 3461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:53.526921 3461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:53.526968 3461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:53.527500 3461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576865c21b004fe7ae49682a0b5b4415" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } }
I20260927 02:52:53.527598 3461 leader_election.cc:302] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 576865c21b004fe7ae49682a0b5b4415; no voters:
I20260927 02:52:53.527737 3461 leader_election.cc:288] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:53.527772 3466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:53.527913 3466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [term 1 LEADER]: Becoming Leader. State: Replica: 576865c21b004fe7ae49682a0b5b4415, State: Running, Role: LEADER
I20260927 02:52:53.527971 3461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:53.528028 3466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576865c21b004fe7ae49682a0b5b4415" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } }
I20260927 02:52:53.528426 3467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "576865c21b004fe7ae49682a0b5b4415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576865c21b004fe7ae49682a0b5b4415" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } } }
I20260927 02:52:53.528532 3467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:53.528429 3468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 576865c21b004fe7ae49682a0b5b4415. Latest consensus state: current_term: 1 leader_uuid: "576865c21b004fe7ae49682a0b5b4415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576865c21b004fe7ae49682a0b5b4415" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } } }
W20260927 02:52:53.529749 3482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:52:53.529865 3482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:52:53.529808 3468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:53.529973 3475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:53.530198 3475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:52:53 dist-test-slave-mfx9 krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:53.531991 3475 catalog_manager.cc:1347] Generated new cluster ID: a9a566eaba3b46ed8fa9b2f8460793fd
I20260927 02:52:53.532047 3475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:53.534292 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:53.523732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.0.98.62:38273)
0927 02:52:53.523881 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:53.523891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:53.523922 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:52:53.524501 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:53.524509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:53.524570 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:53.524717 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:53.525231 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.526565 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.527404 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.527989 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.528147 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:53.530844 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:53.530847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:53.530848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:53.530858 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:53.532604 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:53.532963 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:53.532966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:53.532967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:53.533058 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:53.533212 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:53.533213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:53.533214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:53.533497 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:53.533620 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:53.533983 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 02:52:53.534107 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260927 02:52:53.537948 3475 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:53.538620 3475 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:53.555644 3475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 576865c21b004fe7ae49682a0b5b4415: Generated new TSK 0
I20260927 02:52:53.555836 3475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:52:53.581602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:53.678138 3487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:53.678331 3487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:53.678354 3487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:53.679508 3487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:53.679553 3487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:53.679581 3487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:53.680990 3487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:53.681202 3487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:53.681411 3487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:53.686053 3493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:53.686041 3492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:53.686132 3487 server_base.cc:1034] running on GCE node
W20260927 02:52:53.686041 3495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:53.686460 3487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:53.686676 3487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:53.687805 3487 hybrid_clock.cc:648] HybridClock initialized: now 1790477573687793 us; error 29 us; skew 500 ppm
Sep 27 02:52:53 dist-test-slave-mfx9 krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:53.690021 3487 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:52:53.690388 3487 webserver.cc:460] Webserver started at http://127.0.98.1:44687/ using document root <none> and password file <none>
I20260927 02:52:53.690560 3487 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:53.690598 3487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:53.690683 3487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:53.691715 3487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "82b40689664245f8a5d8173d159866e1"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "05f00e4c8b0a8cf50cd67a3a8e8efc3a"
server_key_iv: "5bc2e89e751f4ed48dbc28a086c6fa1c"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.691998 3487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "82b40689664245f8a5d8173d159866e1"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "05f00e4c8b0a8cf50cd67a3a8e8efc3a"
server_key_iv: "5bc2e89e751f4ed48dbc28a086c6fa1c"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.693128 3487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:52:53.693737 3502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:53.693864 3487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:53.693940 3487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "82b40689664245f8a5d8173d159866e1"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "05f00e4c8b0a8cf50cd67a3a8e8efc3a"
server_key_iv: "5bc2e89e751f4ed48dbc28a086c6fa1c"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.694003 3487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:53.723294 3487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:53.725042 3487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:53.725175 3487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:53.725394 3487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:53.725687 3487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:53.725719 3487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:53.725747 3487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:53.725769 3487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:53.731253 3487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38689
I20260927 02:52:53.731304 3615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38689 every 8 connection(s)
I20260927 02:52:53.731592 3487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:52:53.736538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 3487
I20260927 02:52:53.736627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:53.736817 392 external_mini_cluster.cc:1428] Setting key 2fda2466a120a6df26fc5010a4a4d610
Sep 27 02:52:53 dist-test-slave-mfx9 krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:53.740178 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:53.732229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32879 (local address 127.0.98.62:38273)
0927 02:52:53.732267 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:53.732269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:53.732493 (+ 224us) server_negotiation.cc:407] Connection header received
0927 02:52:53.732992 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:53.732993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:53.733004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:53.733055 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:53.733943 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.734428 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.735417 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.735610 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.735664 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:53.737730 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:53.737731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:53.737731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:53.737737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:53.738915 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:53.739332 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:53.739332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:53.739333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:53.739377 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:53.739614 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:53.739614 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:53.739614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:53.739702 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:53.739753 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:53.740011 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:52:53.740093 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:52:53.740329 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:53.732214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38273 (local address 127.0.98.1:32879)
0927 02:52:53.732376 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:53.732413 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:53.732891 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:53.733126 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:53.733130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:53.733663 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:53.733851 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:53.733856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.734506 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.734507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:53.735333 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:53.735336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.735674 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.735675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:53.735781 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:53.736205 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:53.736212 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:53.737652 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:53.738985 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:53.738991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:53.738993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:53.739267 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:53.739434 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:53.739434 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:53.739435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:53.739562 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:53.739763 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:53.739769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:53.739892 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 02:52:53.739985 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:52:53.740129 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260927 02:52:53.740949 3616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38273
I20260927 02:52:53.741025 3616 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:53.741176 3616 heartbeater.cc:511] Master 127.0.98.62:38273 requested a full tablet report, sending...
I20260927 02:52:53.741668 3425 ts_manager.cc:194] Registered new tserver with Master: 82b40689664245f8a5d8173d159866e1 (127.0.98.1:38689)
I20260927 02:52:53.742238 3425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32879
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:52:53.759490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:53.836526 3621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:53.836711 3621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:53.836733 3621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:53.837879 3621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:53.837920 3621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:53.837949 3621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:53.839345 3621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:53.839541 3621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:53.839728 3621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:53.844211 3626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:53.844303 3627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:53.844307 3621 server_base.cc:1034] running on GCE node
W20260927 02:52:53.844215 3629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:53.844658 3621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:53.844857 3621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:53.845996 3621 hybrid_clock.cc:648] HybridClock initialized: now 1790477573845984 us; error 28 us; skew 500 ppm
Sep 27 02:52:53 dist-test-slave-mfx9 krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:53.848242 3621 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:52:53.848608 3621 webserver.cc:460] Webserver started at http://127.0.98.2:44581/ using document root <none> and password file <none>
I20260927 02:52:53.848783 3621 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:53.848826 3621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:53.848948 3621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:53.849881 3621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "8412159dcb524f8db077b799fe8af1c2"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "e5564b5fe3d6c18591dcbccfcac753e4"
server_key_iv: "aed9a22e3ccf25ce287350a21366b3b7"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.850203 3621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8412159dcb524f8db077b799fe8af1c2"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "e5564b5fe3d6c18591dcbccfcac753e4"
server_key_iv: "aed9a22e3ccf25ce287350a21366b3b7"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.851389 3621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:52:53.851989 3636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:53.852133 3621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:53.852206 3621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "8412159dcb524f8db077b799fe8af1c2"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "e5564b5fe3d6c18591dcbccfcac753e4"
server_key_iv: "aed9a22e3ccf25ce287350a21366b3b7"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.852262 3621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:53.860415 3621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:53.861927 3621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:53.862043 3621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:53.862249 3621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:53.862511 3621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:53.862542 3621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:53.862567 3621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:53.862584 3621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:53.867842 3621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44711
I20260927 02:52:53.867901 3749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44711 every 8 connection(s)
I20260927 02:52:53.868165 3621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:52:53.873129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 3621
I20260927 02:52:53.873236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:53.873401 392 external_mini_cluster.cc:1428] Setting key cf7c6175c9fcebafbbf696e5e0ed79ce
Sep 27 02:52:53 dist-test-slave-mfx9 krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:53.876684 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:53.868719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43191 (local address 127.0.98.62:38273)
0927 02:52:53.868778 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:53.868780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:53.868960 (+ 180us) server_negotiation.cc:407] Connection header received
0927 02:52:53.869433 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:53.869434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:53.869446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:53.869494 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:53.870359 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.870900 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.871834 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.872034 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.872089 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:53.874167 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:53.874168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:53.874168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:53.874175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:53.875512 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:53.875845 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:53.875846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:53.875847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:53.875903 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:53.876110 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:53.876110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:53.876110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:53.876218 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:53.876270 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:53.876521 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 02:52:53.876600 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:52:53.876842 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:53.868688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38273 (local address 127.0.98.2:43191)
0927 02:52:53.868839 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:53.868885 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:53.869341 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:53.869559 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:53.869563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:53.870093 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:53.870272 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:53.870276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.870982 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.870983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:53.871756 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:53.871758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:53.872099 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:53.872100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:53.872211 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:53.872617 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:53.872625 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:53.874093 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:53.875579 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:53.875585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:53.875588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:53.875784 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:53.875954 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:53.875955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:53.875955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:53.876056 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:53.876270 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:53.876275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:53.876405 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:52:53.876490 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:52:53.876632 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260927 02:52:53.877210 3750 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38273
I20260927 02:52:53.877292 3750 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:53.877463 3750 heartbeater.cc:511] Master 127.0.98.62:38273 requested a full tablet report, sending...
I20260927 02:52:53.877799 3425 ts_manager.cc:194] Registered new tserver with Master: 8412159dcb524f8db077b799fe8af1c2 (127.0.98.2:44711)
I20260927 02:52:53.878151 3425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43191
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:52:53.895669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:53.973129 3755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:53.973325 3755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:53.973346 3755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:53.974432 3755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:53.974463 3755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:53.974488 3755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:53.975837 3755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:53.976015 3755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:53.976181 3755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:53.980778 3760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:53.980783 3761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:53.980798 3763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:53.980867 3755 server_base.cc:1034] running on GCE node
I20260927 02:52:53.981199 3755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:53.981426 3755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:53.982566 3755 hybrid_clock.cc:648] HybridClock initialized: now 1790477573982556 us; error 29 us; skew 500 ppm
Sep 27 02:52:53 dist-test-slave-mfx9 krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:53.984881 3755 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:52:53.985296 3755 webserver.cc:460] Webserver started at http://127.0.98.3:46849/ using document root <none> and password file <none>
I20260927 02:52:53.985466 3755 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:53.985505 3755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:53.985592 3755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:53.986534 3755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "21ceae58e61941e485b452bba6344c26"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "f68d51eb1f782cb7b4655b4ccc33213a"
server_key_iv: "35eaec81cf4cfcd7106ee24863f32a40"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.986855 3755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21ceae58e61941e485b452bba6344c26"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "f68d51eb1f782cb7b4655b4ccc33213a"
server_key_iv: "35eaec81cf4cfcd7106ee24863f32a40"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.988060 3755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:53.988660 3770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:53.988809 3755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:53.988906 3755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "21ceae58e61941e485b452bba6344c26"
format_stamp: "Formatted at 2026-09-27 02:52:53 on dist-test-slave-mfx9"
server_key: "f68d51eb1f782cb7b4655b4ccc33213a"
server_key_iv: "35eaec81cf4cfcd7106ee24863f32a40"
server_key_version: "encryptionkey@0"
I20260927 02:52:53.988960 3755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:54.021493 3755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:54.023474 3755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:54.023584 3755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:54.023767 3755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:54.024039 3755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:54.024067 3755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:54.024087 3755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:54.024101 3755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:54.029352 3755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46117
I20260927 02:52:54.029443 3883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46117 every 8 connection(s)
I20260927 02:52:54.029701 3755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:52:54.030009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 3755
I20260927 02:52:54.030104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:54.030265 392 external_mini_cluster.cc:1428] Setting key dca77bc13552069d9e4f7166e6190b10
Sep 27 02:52:54 dist-test-slave-mfx9 krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:54.038168 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:54.030405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51869 (local address 127.0.98.62:38273)
0927 02:52:54.030452 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:54.030454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:54.030666 (+ 212us) server_negotiation.cc:407] Connection header received
0927 02:52:54.031257 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:54.031257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:54.031267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:54.031304 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:54.032422 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:54.032899 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:54.033833 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:54.034040 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:54.034093 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:54.035972 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:54.035973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:54.035973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:54.035981 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:54.037009 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:54.037334 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:54.037335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:54.037336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:54.037383 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:54.037599 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:54.037600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:54.037600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:54.037701 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:54.037753 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:54.038000 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:52:54.038083 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:52:54.038321 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:54.030372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38273 (local address 127.0.98.3:51869)
0927 02:52:54.030535 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:54.030580 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:54.031178 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:54.031367 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:54.031370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:54.032074 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:54.032324 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:54.032330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:54.032976 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:54.032976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:54.033758 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:54.033760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:54.034103 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:54.034103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:54.034231 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:54.034620 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:54.034627 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:54.035886 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:54.037074 (+ 1188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:54.037080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:54.037082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:54.037273 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:54.037445 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:54.037446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:54.037446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:54.037546 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:54.037752 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:54.037757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:54.037882 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 02:52:54.037973 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:52:54.038115 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260927 02:52:54.038713 3884 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38273
I20260927 02:52:54.038817 3884 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:54.038966 3884 heartbeater.cc:511] Master 127.0.98.62:38273 requested a full tablet report, sending...
I20260927 02:52:54.039283 3425 ts_manager.cc:194] Registered new tserver with Master: 21ceae58e61941e485b452bba6344c26 (127.0.98.3:46117)
I20260927 02:52:54.039633 3425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51869
I20260927 02:52:54.041651 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:52:54 dist-test-slave-mfx9 krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477574, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 02:52:54 dist-test-slave-mfx9 krb5kdc[3380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477574, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:54.061916 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:54.055527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.0.98.62:38273)
0927 02:52:54.055605 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:54.055606 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:54.055673 (+ 67us) server_negotiation.cc:407] Connection header received
0927 02:52:54.055771 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:54.055772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:54.055783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:54.055832 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:54.056356 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:54.056832 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:54.057605 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:54.057795 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:54.057847 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:54.059802 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:54.059803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:54.059803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:54.059810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:54.060909 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:54.061220 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:54.061220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:54.061221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:54.061269 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:54.061429 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:54.061430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:54.061431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:54.061571 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:54.061657 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:54.061751 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:52:54.061827 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:52:54.062129 3425 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58728
I20260927 02:52:54.062475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 3487
I20260927 02:52:54.066366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 3621
I20260927 02:52:54.069809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 3755
I20260927 02:52:54.073277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 3393
2026-09-27T02:52:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1716 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:54 dist-test-slave-mfx9 krb5kdc[3896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:54 dist-test-slave-mfx9 krb5kdc[3896](info): set up 2 sockets
Sep 27 02:52:54 dist-test-slave-mfx9 krb5kdc[3896](info): commencing operation
krb5kdc: starting...
W20260927 02:52:55.006280 392 mini_kdc.cc:121] Time spent starting KDC: real 0.915s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:52:55.116707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:55.196246 3909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:55.196461 3909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:55.196493 3909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:55.197625 3909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:55.197664 3909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:55.197682 3909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:55.197697 3909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:55.197713 3909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:55.199198 3909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:55.199431 3909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:55.199620 3909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:55.204100 3914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:55.204146 3917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:55.204071 3915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:55.204361 3909 server_base.cc:1034] running on GCE node
I20260927 02:52:55.204572 3909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:55.204802 3909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:55.205933 3909 hybrid_clock.cc:648] HybridClock initialized: now 1790477575205919 us; error 28 us; skew 500 ppm
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:55.208199 3909 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:55.208612 3909 webserver.cc:460] Webserver started at http://127.0.98.62:46167/ using document root <none> and password file <none>
I20260927 02:52:55.208779 3909 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:55.208817 3909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:55.208904 3909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:55.209857 3909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "76ff57ae94df43738ce57a1b010fd34b"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "b397737914aa50a678fecee89910ba0e"
server_key_iv: "14cd243b2054c55c9650ae02d9fd8d64"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.210142 3909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "76ff57ae94df43738ce57a1b010fd34b"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "b397737914aa50a678fecee89910ba0e"
server_key_iv: "14cd243b2054c55c9650ae02d9fd8d64"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.211335 3909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:55.211946 3924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.212083 3909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:55.212158 3909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "76ff57ae94df43738ce57a1b010fd34b"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "b397737914aa50a678fecee89910ba0e"
server_key_iv: "14cd243b2054c55c9650ae02d9fd8d64"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.212216 3909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:55.223202 3909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:55.224879 3909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:55.224997 3909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:55.228542 3909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41489
I20260927 02:52:55.228572 3976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41489 every 8 connection(s)
I20260927 02:52:55.228863 3909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:55.229609 3977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:55.230649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 3909
I20260927 02:52:55.230738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:55.230921 392 external_mini_cluster.cc:1428] Setting key 99bd59533e807a8c52d4e4c2b33a9024
I20260927 02:52:55.232045 3977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b: Bootstrap starting.
I20260927 02:52:55.232715 3977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:55.232982 3977 log.cc:824] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:55.233603 3977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b: No bootstrap required, opened a new log
I20260927 02:52:55.235802 3977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ff57ae94df43738ce57a1b010fd34b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } }
I20260927 02:52:55.236002 3977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:55.236037 3977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ff57ae94df43738ce57a1b010fd34b, State: Initialized, Role: FOLLOWER
I20260927 02:52:55.236176 3977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ff57ae94df43738ce57a1b010fd34b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } }
I20260927 02:52:55.236269 3977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:55.236310 3977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:55.236354 3977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:55.236967 3977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ff57ae94df43738ce57a1b010fd34b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } }
I20260927 02:52:55.237053 3977 leader_election.cc:302] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76ff57ae94df43738ce57a1b010fd34b; no voters:
I20260927 02:52:55.237200 3977 leader_election.cc:288] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:55.237232 3982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:55.237375 3982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [term 1 LEADER]: Becoming Leader. State: Replica: 76ff57ae94df43738ce57a1b010fd34b, State: Running, Role: LEADER
I20260927 02:52:55.237450 3977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:55.237542 3982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ff57ae94df43738ce57a1b010fd34b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } }
I20260927 02:52:55.237862 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76ff57ae94df43738ce57a1b010fd34b. Latest consensus state: current_term: 1 leader_uuid: "76ff57ae94df43738ce57a1b010fd34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ff57ae94df43738ce57a1b010fd34b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } }
I20260927 02:52:55.238101 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:55.238039 3984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76ff57ae94df43738ce57a1b010fd34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ff57ae94df43738ce57a1b010fd34b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } }
I20260927 02:52:55.238484 3984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:55.238627 3991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:55.239091 3991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:55.240578 3991 catalog_manager.cc:1347] Generated new cluster ID: c195bb8d58d44972ba416f374539af5c
I20260927 02:52:55.240624 3991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:55.241590 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.231377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.0.98.62:41489)
0927 02:52:55.231514 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:55.231521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:55.231545 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:52:55.231938 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:55.231945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:55.231989 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:55.232095 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:55.232803 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.233818 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.234826 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.235197 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.235273 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.237804 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:55.237807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:55.237808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:55.237818 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:55.239739 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.240088 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.240091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.240091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.240180 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.240365 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.240367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.240367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.240673 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:55.240796 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:55.241263 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 02:52:55.241391 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260927 02:52:55.258354 3991 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:55.259100 3991 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:55.264796 3991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76ff57ae94df43738ce57a1b010fd34b: Generated new TSK 0
I20260927 02:52:55.264961 3991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:52:55.293079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:55.371093 4003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:55.371275 4003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:55.371296 4003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:55.372414 4003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:55.372453 4003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:55.372480 4003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:55.373893 4003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:55.374099 4003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:55.374302 4003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:55.378883 4008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:55.378980 4003 server_base.cc:1034] running on GCE node
W20260927 02:52:55.378890 4009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:55.378963 4011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:55.379328 4003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:55.379549 4003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:55.380681 4003 hybrid_clock.cc:648] HybridClock initialized: now 1790477575380664 us; error 31 us; skew 500 ppm
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:55.382961 4003 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:52:55.383431 4003 webserver.cc:460] Webserver started at http://127.0.98.1:37841/ using document root <none> and password file <none>
I20260927 02:52:55.383634 4003 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:55.383679 4003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:55.383800 4003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:55.384764 4003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d1032591315431b94c76f8285b021a3"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "b9a7bbaf7faec127ba20eee70cf81e55"
server_key_iv: "5f8b2885b9189c792867f91d8689b41d"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.385085 4003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d1032591315431b94c76f8285b021a3"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "b9a7bbaf7faec127ba20eee70cf81e55"
server_key_iv: "5f8b2885b9189c792867f91d8689b41d"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.386219 4003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:55.386875 4018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.387063 4003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.387141 4003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "6d1032591315431b94c76f8285b021a3"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "b9a7bbaf7faec127ba20eee70cf81e55"
server_key_iv: "5f8b2885b9189c792867f91d8689b41d"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.387203 4003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:55.420616 4003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:55.422581 4003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:55.422709 4003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:55.422953 4003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:55.423247 4003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:55.423280 4003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.423308 4003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:55.423332 4003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.428457 4003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40165
I20260927 02:52:55.428498 4131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40165 every 8 connection(s)
I20260927 02:52:55.428778 4003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:55.436638 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.429501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49145 (local address 127.0.98.62:41489)
0927 02:52:55.429551 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:55.429553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:55.429739 (+ 186us) server_negotiation.cc:407] Connection header received
0927 02:52:55.430629 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:55.430629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:55.430638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:55.430675 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:55.431252 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.431740 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.432624 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.432809 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.432850 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.434608 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:55.434609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:55.434609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:55.434615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:55.435669 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.435991 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.435992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.435992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.436033 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.436196 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.436196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.436196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.436283 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:55.436325 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:55.436520 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:52:55.436577 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:52:55.436795 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.429477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.98.1:49145)
0927 02:52:55.429623 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:55.429663 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:55.430533 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:55.430737 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:55.430740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:55.431012 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:55.431191 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:55.431196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.431798 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.431799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:55.432569 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:55.432570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.432857 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.432858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:55.432949 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.433309 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:55.433316 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:55.434530 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:55.435729 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:55.435733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:55.435735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:55.435936 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:55.436075 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:55.436075 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:55.436076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:55.436156 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:55.436321 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:55.436324 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:55.436431 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 02:52:55.436503 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:52:55.436608 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260927 02:52:55.437216 4132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41489
I20260927 02:52:55.437305 4132 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:55.437440 4132 heartbeater.cc:511] Master 127.0.98.62:41489 requested a full tablet report, sending...
I20260927 02:52:55.437657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 4003
I20260927 02:52:55.437779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:55.437947 392 external_mini_cluster.cc:1428] Setting key 938d91855584eb0d900ac4cd26d2347f
I20260927 02:52:55.437956 3941 ts_manager.cc:194] Registered new tserver with Master: 6d1032591315431b94c76f8285b021a3 (127.0.98.1:40165)
I20260927 02:52:55.438627 3941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49145
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:52:55.461089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:55.538163 4137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:55.538360 4137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:55.538385 4137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:55.539647 4137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:55.539690 4137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:55.539714 4137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:55.541136 4137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:55.541338 4137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:55.541565 4137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:55.546098 4143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:55.546105 4145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:55.546118 4142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:55.546303 4137 server_base.cc:1034] running on GCE node
I20260927 02:52:55.546551 4137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:55.546785 4137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:55.547930 4137 hybrid_clock.cc:648] HybridClock initialized: now 1790477575547919 us; error 31 us; skew 500 ppm
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:55.550264 4137 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:52:55.550662 4137 webserver.cc:460] Webserver started at http://127.0.98.2:38197/ using document root <none> and password file <none>
I20260927 02:52:55.550877 4137 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:55.550925 4137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:55.551038 4137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:55.552034 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "f32fe8102d8149309614f0c7d1c1b71c"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "2fb897102eeba34235ef888073f95a0f"
server_key_iv: "adb629f6903ed6a9cb79041ed2d73c74"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.552361 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f32fe8102d8149309614f0c7d1c1b71c"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "2fb897102eeba34235ef888073f95a0f"
server_key_iv: "adb629f6903ed6a9cb79041ed2d73c74"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.553581 4137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:55.554189 4152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.554337 4137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:55.554440 4137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "f32fe8102d8149309614f0c7d1c1b71c"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "2fb897102eeba34235ef888073f95a0f"
server_key_iv: "adb629f6903ed6a9cb79041ed2d73c74"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.554497 4137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:55.573098 4137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:55.575423 4137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:55.575569 4137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:55.575789 4137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:55.576082 4137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:55.576114 4137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.576155 4137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:55.576180 4137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.581661 4137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37269
I20260927 02:52:55.581735 4265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37269 every 8 connection(s)
I20260927 02:52:55.582044 4137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:52:55.585283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 4137
I20260927 02:52:55.585366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:55.585512 392 external_mini_cluster.cc:1428] Setting key 0592bd3a04c189681fc5a2aa59d37025
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:55.591256 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.582749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51771 (local address 127.0.98.62:41489)
0927 02:52:55.582828 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:55.582831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:55.583052 (+ 221us) server_negotiation.cc:407] Connection header received
0927 02:52:55.583951 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:55.583952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:55.583962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:55.584019 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:55.584578 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.585070 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.586172 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.586446 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.586509 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.588659 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:55.588660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:55.588660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:55.588668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:55.589899 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.590270 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.590272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.590272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.590339 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.590605 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.590605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.590606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.590748 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:55.590846 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:55.591079 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:52:55.591163 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:52:55.591419 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.582726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.98.2:51771)
0927 02:52:55.582921 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:55.582957 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:55.583866 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:55.584073 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:55.584076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:55.584325 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:55.584519 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:55.584523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.585132 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.585132 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:55.586099 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:55.586102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.586521 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.586522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:55.586622 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.587098 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:55.587106 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:55.588577 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:55.589975 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:55.589981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:55.589983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:55.590189 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:55.590421 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:55.590421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:55.590422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:55.590553 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:55.590833 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:55.590836 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:55.590959 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 02:52:55.591038 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:52:55.591155 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260927 02:52:55.591830 4266 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41489
I20260927 02:52:55.591907 4266 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:55.592082 4266 heartbeater.cc:511] Master 127.0.98.62:41489 requested a full tablet report, sending...
I20260927 02:52:55.592466 3941 ts_manager.cc:194] Registered new tserver with Master: f32fe8102d8149309614f0c7d1c1b71c (127.0.98.2:37269)
I20260927 02:52:55.592787 3941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51771
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:52:55.609035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:55.687685 4271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:55.687893 4271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:55.687914 4271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:55.689003 4271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:55.689035 4271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:55.689059 4271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:55.690505 4271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:55.690726 4271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:55.690965 4271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:55.695535 4279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:55.695534 4276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:55.695782 4271 server_base.cc:1034] running on GCE node
W20260927 02:52:55.695809 4277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:55.696089 4271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:55.696328 4271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:55.697476 4271 hybrid_clock.cc:648] HybridClock initialized: now 1790477575697463 us; error 28 us; skew 500 ppm
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:55.705572 4271 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:52:55.706043 4271 webserver.cc:460] Webserver started at http://127.0.98.3:44227/ using document root <none> and password file <none>
I20260927 02:52:55.706274 4271 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:55.706323 4271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:55.706450 4271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:55.707510 4271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "be9156748a4f4069ba3c8ad9c8ae0b45"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "c8e56cd4c6787c47d38918a121670aad"
server_key_iv: "4227ef247caaed411b0b5cd24d18b299"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.707844 4271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be9156748a4f4069ba3c8ad9c8ae0b45"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "c8e56cd4c6787c47d38918a121670aad"
server_key_iv: "4227ef247caaed411b0b5cd24d18b299"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.708995 4271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:52:55.709613 4286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.709755 4271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:55.709832 4271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "be9156748a4f4069ba3c8ad9c8ae0b45"
format_stamp: "Formatted at 2026-09-27 02:52:55 on dist-test-slave-mfx9"
server_key: "c8e56cd4c6787c47d38918a121670aad"
server_key_iv: "4227ef247caaed411b0b5cd24d18b299"
server_key_version: "encryptionkey@0"
I20260927 02:52:55.709889 4271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:55.723965 4271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:55.725600 4271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:55.725733 4271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:55.725934 4271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:55.726209 4271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:55.726249 4271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.726276 4271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:55.726293 4271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:55.731606 4271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36873
I20260927 02:52:55.731642 4399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36873 every 8 connection(s)
I20260927 02:52:55.731927 4271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:52:55.733294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 4271
I20260927 02:52:55.733402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:55.733599 392 external_mini_cluster.cc:1428] Setting key e2cf46feec52566df9a3328b0b4d2087
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:55.740525 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.732552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45627 (local address 127.0.98.62:41489)
0927 02:52:55.732609 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:55.732611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:55.732829 (+ 218us) server_negotiation.cc:407] Connection header received
0927 02:52:55.733858 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:55.733859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:55.733871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:55.733929 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:55.734621 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.735164 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.736080 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.736276 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.736327 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.738278 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:55.738279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:55.738280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:55.738286 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:55.739534 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.739857 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.739858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.739858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.739906 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.740063 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.740063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.740064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.740154 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:55.740203 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:55.740397 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 02:52:55.740451 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:52:55.740677 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.732533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.98.3:45627)
0927 02:52:55.732679 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:55.732719 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:55.733763 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:55.734003 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:55.734007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:55.734330 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:55.734551 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:55.734556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.735226 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.735226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:55.736009 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:55.736011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.736334 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.736334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:55.736423 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.736840 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:55.736850 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:55.738189 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:55.739598 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:55.739603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:55.739605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:55.739801 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:55.739944 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:55.739945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:55.739945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:55.740024 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:55.740198 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:55.740201 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:55.740305 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 02:52:55.740378 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:52:55.740502 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260927 02:52:55.741067 4400 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41489
I20260927 02:52:55.741142 4400 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:55.741312 4400 heartbeater.cc:511] Master 127.0.98.62:41489 requested a full tablet report, sending...
I20260927 02:52:55.741644 3941 ts_manager.cc:194] Registered new tserver with Master: be9156748a4f4069ba3c8ad9c8ae0b45 (127.0.98.3:36873)
I20260927 02:52:55.742033 3941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45627
I20260927 02:52:55.745101 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:52:55.766098 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.758959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.0.98.1:40165)
0927 02:52:55.759150 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:55.759155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:55.759166 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:52:55.759195 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:55.759197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:55.759211 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:55.759294 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:55.759805 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.760543 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.761284 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.761591 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.761645 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.763670 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:55.763677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:55.763677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:55.763684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:55.765166 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.765448 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.765450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.765451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.765500 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.765626 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.765626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.765627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.765736 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:55.765860 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:55.765949 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:52:55.766006 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[3896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:55.773062 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:55.767256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.0.98.62:41489)
0927 02:52:55.767310 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:55.767312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:55.767320 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:55.767361 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:55.767362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:55.767370 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:55.767412 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:55.767873 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.768340 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.769076 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:55.769253 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:55.769296 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:55.771214 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:55.771215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:55.771216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:55.771222 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:55.772231 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.772513 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.772514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.772514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.772556 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:55.772678 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:55.772678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:55.772679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:55.772795 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:55.772862 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:55.772933 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:52:55.772987 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 02:52:55.773285 3941 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52530
I20260927 02:52:55.773588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 4003
I20260927 02:52:55.777450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 4137
I20260927 02:52:55.781105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 4271
I20260927 02:52:55.784736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 3909
2026-09-27T02:52:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1711 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[4412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[4412](info): set up 2 sockets
Sep 27 02:52:55 dist-test-slave-mfx9 krb5kdc[4412](info): commencing operation
krb5kdc: starting...
W20260927 02:52:56.726673 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:52:56 dist-test-slave-mfx9 krb5kdc[4412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:52:56.835561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40047
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:56.913137 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:56.913329 4425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:56.913349 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:56.914463 4425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:56.914501 4425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:56.914515 4425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:56.914525 4425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:56.914535 4425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:56.915994 4425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:56.916205 4425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:56.916409 4425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:56.920936 4433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:56.920958 4425 server_base.cc:1034] running on GCE node
W20260927 02:52:56.920979 4430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:56.920943 4431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:56.921415 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:56.921674 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:56.922820 4425 hybrid_clock.cc:648] HybridClock initialized: now 1790477576922801 us; error 31 us; skew 500 ppm
Sep 27 02:52:56 dist-test-slave-mfx9 krb5kdc[4412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:56.925189 4425 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:52:56.925493 4425 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:56.925541 4425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:56.925659 4425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:56.926702 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "65c517fe9d73442f9d30b398c370de40"
format_stamp: "Formatted at 2026-09-27 02:52:56 on dist-test-slave-mfx9"
server_key: "c2be688c5a36bc1ce4ae838b2efa2474"
server_key_iv: "6b64d222d8da831633bbbe4d8ae95178"
server_key_version: "encryptionkey@0"
I20260927 02:52:56.927057 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "65c517fe9d73442f9d30b398c370de40"
format_stamp: "Formatted at 2026-09-27 02:52:56 on dist-test-slave-mfx9"
server_key: "c2be688c5a36bc1ce4ae838b2efa2474"
server_key_iv: "6b64d222d8da831633bbbe4d8ae95178"
server_key_version: "encryptionkey@0"
I20260927 02:52:56.928191 4425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:56.928786 4439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:56.928929 4425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:56.929006 4425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "65c517fe9d73442f9d30b398c370de40"
format_stamp: "Formatted at 2026-09-27 02:52:56 on dist-test-slave-mfx9"
server_key: "c2be688c5a36bc1ce4ae838b2efa2474"
server_key_iv: "6b64d222d8da831633bbbe4d8ae95178"
server_key_version: "encryptionkey@0"
I20260927 02:52:56.929061 4425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:56.939190 4425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:56.940775 4425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:56.940903 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:56.944296 4425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40047
I20260927 02:52:56.944315 4491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40047 every 8 connection(s)
I20260927 02:52:56.944551 4425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:56.945348 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:56.947440 4492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40: Bootstrap starting.
I20260927 02:52:56.947952 4492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:56.948191 4492 log.cc:824] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:56.948729 4492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40: No bootstrap required, opened a new log
I20260927 02:52:56.949684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 4425
I20260927 02:52:56.949788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:56.949951 392 external_mini_cluster.cc:1428] Setting key e89442a6701c9636ce84a9a104d00e5e
I20260927 02:52:56.950213 4492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c517fe9d73442f9d30b398c370de40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260927 02:52:56.950352 4492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:56.950384 4492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c517fe9d73442f9d30b398c370de40, State: Initialized, Role: FOLLOWER
I20260927 02:52:56.950548 4492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c517fe9d73442f9d30b398c370de40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260927 02:52:56.950632 4492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:56.950662 4492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:56.950703 4492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:56.951567 4492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c517fe9d73442f9d30b398c370de40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260927 02:52:56.951665 4492 leader_election.cc:302] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c517fe9d73442f9d30b398c370de40; no voters:
I20260927 02:52:56.951794 4492 leader_election.cc:288] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:56.951840 4497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:56.951987 4497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [term 1 LEADER]: Becoming Leader. State: Replica: 65c517fe9d73442f9d30b398c370de40, State: Running, Role: LEADER
I20260927 02:52:56.952008 4492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:56.952848 4497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c517fe9d73442f9d30b398c370de40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
W20260927 02:52:56.953634 4511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 02:52:56.954540 4499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c517fe9d73442f9d30b398c370de40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c517fe9d73442f9d30b398c370de40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } } }
I20260927 02:52:56.954671 4498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c517fe9d73442f9d30b398c370de40. Latest consensus state: current_term: 1 leader_uuid: "65c517fe9d73442f9d30b398c370de40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c517fe9d73442f9d30b398c370de40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } } }
W20260927 02:52:56.954891 4511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:52:56.954917 4498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:56.955072 4499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:56.955770 4514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:56.955940 4514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:52:56 dist-test-slave-mfx9 krb5kdc[4412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:56.957681 4514 catalog_manager.cc:1347] Generated new cluster ID: 9357acc30dc646c38ae989f252479be5
I20260927 02:52:56.957741 4514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:56.960978 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:56.950445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.0.98.62:40047)
0927 02:52:56.950604 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:56.950612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:56.950641 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:52:56.950747 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:56.950774 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:56.950828 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:56.950986 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:56.951538 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:56.952727 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:56.953740 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:56.954128 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:56.954228 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:56.956800 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:56.956803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:56.956804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:56.956816 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:56.959471 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:56.959822 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:56.959825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:56.959825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:56.959931 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:56.960118 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:56.960119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:56.960119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:56.960411 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:56.960581 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:56.960658 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:52:56.960788 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260927 02:52:56.978022 4514 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:56.978525 4514 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:56.985304 4514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40: Generated new TSK 0
I20260927 02:52:56.985466 4514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:52:57.009469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:57.087556 4518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:57.087759 4518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:57.087781 4518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:57.088913 4518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:57.088951 4518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:57.088980 4518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:57.090467 4518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:57.090691 4518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:57.090945 4518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:57.095923 4526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:57.095923 4523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:57.095942 4524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:57.096094 4518 server_base.cc:1034] running on GCE node
I20260927 02:52:57.096386 4518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:57.096582 4518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:57.097718 4518 hybrid_clock.cc:648] HybridClock initialized: now 1790477577097701 us; error 30 us; skew 500 ppm
Sep 27 02:52:57 dist-test-slave-mfx9 krb5kdc[4412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:57.100189 4518 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:52:57.100523 4518 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:57.100574 4518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:57.100694 4518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:57.101698 4518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "95c77e54a9074de6bbd27e4867babc41"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "4d583191f83155c6c2e13d82ed51a9d8"
server_key_iv: "bdf62fb94190fc7e3e71c95feb06ecd8"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.102013 4518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "95c77e54a9074de6bbd27e4867babc41"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "4d583191f83155c6c2e13d82ed51a9d8"
server_key_iv: "bdf62fb94190fc7e3e71c95feb06ecd8"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.103235 4518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:57.103871 4532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.104012 4518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:57.104089 4518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "95c77e54a9074de6bbd27e4867babc41"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "4d583191f83155c6c2e13d82ed51a9d8"
server_key_iv: "bdf62fb94190fc7e3e71c95feb06ecd8"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.104146 4518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:57.117753 4518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:57.119953 4518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:57.120086 4518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:57.120302 4518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:57.120595 4518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:57.120627 4518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.120656 4518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:57.120679 4518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.125946 4518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43539
I20260927 02:52:57.125990 4645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43539 every 8 connection(s)
I20260927 02:52:57.126227 4518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:52:57 dist-test-slave-mfx9 krb5kdc[4412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:57.134021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 4518
I20260927 02:52:57.134131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:57.134320 392 external_mini_cluster.cc:1428] Setting key 67721bbbd21b7fece8cb17a8c77b83f2
I20260927 02:52:57.135085 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.126847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36161 (local address 127.0.98.62:40047)
0927 02:52:57.126899 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.126901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.127118 (+ 217us) server_negotiation.cc:407] Connection header received
0927 02:52:57.127625 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.127627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.127639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.127691 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:57.128626 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.129134 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.130125 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.130340 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.130396 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.132442 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:57.132444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:57.132444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:57.132451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:57.133650 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:57.134026 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:57.134028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:57.134028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:57.134095 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:57.134358 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:57.134359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:57.134360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:57.134513 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:57.134580 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.134899 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.134997 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:52:57.135257 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.126865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:36161)
0927 02:52:57.127007 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.127043 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:57.127530 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:57.127760 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:57.127764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:57.128049 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:57.128533 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.128537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.129219 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.129220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.130043 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.130045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.130408 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.130409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.130527 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.131028 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:57.131036 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:57.132351 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:57.133714 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:57.133720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:57.133722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:57.133945 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:57.134180 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:57.134181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:57.134181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:57.134297 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:57.134580 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:57.134585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:57.134741 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 02:52:57.134877 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 02:52:57.135053 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260927 02:52:57.135809 4646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260927 02:52:57.135892 4646 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:57.136035 4646 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260927 02:52:57.136514 4456 ts_manager.cc:194] Registered new tserver with Master: 95c77e54a9074de6bbd27e4867babc41 (127.0.98.1:43539)
I20260927 02:52:57.137094 4456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36161
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:52:57.157380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:57.236812 4651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:57.236999 4651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:57.237020 4651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:57.238145 4651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:57.238183 4651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:57.238207 4651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:57.239687 4651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:57.239883 4651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:57.240082 4651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:57.244654 4657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:57.244642 4656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:57.244628 4659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:57.244786 4651 server_base.cc:1034] running on GCE node
I20260927 02:52:57.245044 4651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:57.245223 4651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:57.246347 4651 hybrid_clock.cc:648] HybridClock initialized: now 1790477577246326 us; error 38 us; skew 500 ppm
Sep 27 02:52:57 dist-test-slave-mfx9 krb5kdc[4412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:57.248663 4651 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:52:57.248931 4651 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:57.248977 4651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:57.249063 4651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:57.249987 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "c7e14f7d83be41cb9d4e45a4714e3c96"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "f2ae815d66f2346d1a32ddeda00c33eb"
server_key_iv: "4adc8a4a16f74103717c586f38c3abc1"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.250255 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c7e14f7d83be41cb9d4e45a4714e3c96"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "f2ae815d66f2346d1a32ddeda00c33eb"
server_key_iv: "4adc8a4a16f74103717c586f38c3abc1"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.251444 4651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:52:57.252041 4665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.252197 4651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:57.252266 4651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "c7e14f7d83be41cb9d4e45a4714e3c96"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "f2ae815d66f2346d1a32ddeda00c33eb"
server_key_iv: "4adc8a4a16f74103717c586f38c3abc1"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.252332 4651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:57.265471 4651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:57.267192 4651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:57.267304 4651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:57.267498 4651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:57.267768 4651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:57.267798 4651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.267818 4651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:57.267841 4651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.273082 4651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34177
I20260927 02:52:57.273131 4778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34177 every 8 connection(s)
I20260927 02:52:57.273360 4651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:52:57 dist-test-slave-mfx9 krb5kdc[4412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:57.281672 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.274051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39289 (local address 127.0.98.62:40047)
0927 02:52:57.274115 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.274117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.274277 (+ 160us) server_negotiation.cc:407] Connection header received
0927 02:52:57.274733 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.274734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.274747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.274813 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:57.275725 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.276218 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.277210 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.277391 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.277438 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.279440 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:57.279442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:57.279442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:57.279448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:57.280633 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:57.280969 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:57.280970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:57.280970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:57.281014 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:57.281194 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:57.281195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:57.281195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:57.281291 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:57.281338 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.281544 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.281608 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:52:57.281770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 4651
I20260927 02:52:57.281900 4781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.274014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:39289)
0927 02:52:57.274157 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.274192 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:57.274658 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:57.274910 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:57.274913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:57.275161 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:57.275654 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.275660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.276297 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.276298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.277148 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.277150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.277452 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.277453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.277557 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.278023 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:57.278031 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:57.279353 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:57.280706 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:57.280711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:57.280713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:57.280912 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:57.281060 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:57.281060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:57.281061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:57.281149 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:57.281339 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:57.281342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:57.281451 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:52:57.281528 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:52:57.281677 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260927 02:52:57.281934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:57.282140 392 external_mini_cluster.cc:1428] Setting key d884ab774cd81e473018f7c78a2619c1
I20260927 02:52:57.282352 4779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260927 02:52:57.282446 4779 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:57.282586 4779 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260927 02:52:57.282950 4456 ts_manager.cc:194] Registered new tserver with Master: c7e14f7d83be41cb9d4e45a4714e3c96 (127.0.98.2:34177)
I20260927 02:52:57.283390 4456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39289
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:52:57.303601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:52:57.381125 4784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:57.381315 4784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:57.381337 4784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:57.382493 4784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:57.382534 4784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:57.382558 4784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:57.384027 4784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:57.384230 4784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:57.384425 4784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:57.388926 4789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:57.388897 4790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:57.389003 4784 server_base.cc:1034] running on GCE node
W20260927 02:52:57.388887 4792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:57.389317 4784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:57.389496 4784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:57.390622 4784 hybrid_clock.cc:648] HybridClock initialized: now 1790477577390608 us; error 26 us; skew 500 ppm
Sep 27 02:52:57 dist-test-slave-mfx9 krb5kdc[4412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:52:57.392889 4784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:52:57.393167 4784 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:57.393213 4784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:57.393332 4784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:57.394294 4784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "90a3d26dc0ad47ce912f86fa53414734"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "0f2197cef949d6c8bbed0cc912afae94"
server_key_iv: "23a55ae8cdd404672d711c7281baac6a"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.394603 4784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "90a3d26dc0ad47ce912f86fa53414734"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "0f2197cef949d6c8bbed0cc912afae94"
server_key_iv: "23a55ae8cdd404672d711c7281baac6a"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.395753 4784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:52:57.396349 4798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.396489 4784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.396565 4784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "90a3d26dc0ad47ce912f86fa53414734"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "0f2197cef949d6c8bbed0cc912afae94"
server_key_iv: "23a55ae8cdd404672d711c7281baac6a"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.396622 4784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:57.408598 4784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:57.410252 4784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:57.410377 4784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:57.410573 4784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:57.410882 4784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:57.410916 4784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.410944 4784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:57.410967 4784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.416190 4784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32967
I20260927 02:52:57.416242 4911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32967 every 8 connection(s)
I20260927 02:52:57.416477 4784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:52:57.417201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 4784
I20260927 02:52:57.417305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:57.417483 392 external_mini_cluster.cc:1428] Setting key 250bbde4d363fce291c726e3388584be
Sep 27 02:52:57 dist-test-slave-mfx9 krb5kdc[4412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:52:57.424733 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.417092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36659 (local address 127.0.98.62:40047)
0927 02:52:57.417157 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.417159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.417306 (+ 147us) server_negotiation.cc:407] Connection header received
0927 02:52:57.417844 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.417845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.417855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.417906 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:57.418917 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.419390 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.420403 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.420606 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.420652 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.422531 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:57.422532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:57.422532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:57.422539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:57.423749 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:57.424074 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:57.424074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:57.424075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:57.424119 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:57.424298 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:57.424298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:57.424299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:57.424386 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:57.424429 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.424620 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.424673 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:52:57.424973 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.417047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:36659)
0927 02:52:57.417187 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.417227 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:57.417749 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:57.417980 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:57.417984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:57.418321 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:57.418842 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.418850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.419491 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.419492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.420331 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.420333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.420668 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.420669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.420801 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.421190 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:52:57.421198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:57.422451 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:57.423817 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:57.423822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:57.423824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:57.424018 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:57.424161 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:52:57.424161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:52:57.424161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:52:57.424256 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:52:57.424425 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:57.424427 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:57.424530 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 02:52:57.424609 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:52:57.424719 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260927 02:52:57.425357 4912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260927 02:52:57.425436 4912 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:57.425570 4912 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260927 02:52:57.425875 4456 ts_manager.cc:194] Registered new tserver with Master: 90a3d26dc0ad47ce912f86fa53414734 (127.0.98.3:32967)
I20260927 02:52:57.426206 4456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36659
I20260927 02:52:57.428960 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:52:57.435220 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.430094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.0.98.62:40047)
0927 02:52:57.430125 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.430127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.430233 (+ 106us) server_negotiation.cc:407] Connection header received
0927 02:52:57.430290 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.430291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.430300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.430338 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:57.430851 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.431366 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.432044 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.432235 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.432286 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.433286 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:57.433288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:57.433288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:52:57.433294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:57.434259 (+ 965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:57.434544 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:57.434545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:57.434545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:57.434593 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:52:57.434745 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:52:57.434745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:52:57.434746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:52:57.434894 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:57.434983 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.435075 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.435140 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:52:57.436502 4456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:52:57.437942 4456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:57.443789 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.439911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43539 (local address 127.0.0.1:60424)
0927 02:52:57.440265 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.440274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:57.440321 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:57.440553 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:57.440554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:57.440557 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:57.440600 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.440601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.441933 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.441934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.442627 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.442628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.443523 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.443524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.443593 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.443608 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 02:52:57.443668 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:52:57.443719 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":84,"threads_started":1}
I20260927 02:52:57.443953 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.439876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32967 (local address 127.0.0.1:54836)
0927 02:52:57.440023 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.440057 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:57.440126 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:57.440544 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:57.440545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:57.440547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:57.440584 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.440586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.441885 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.441886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.442941 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.442943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.443734 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.443735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.443798 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.443801 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:57.443826 (+ 25us) client_negotiation.cc:240] Negotiation successful
0927 02:52:57.443855 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":50,"threads_started":1}
I20260927 02:52:57.443908 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.439902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34177 (local address 127.0.0.1:58200)
0927 02:52:57.440218 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.440225 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:57.440271 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:57.440478 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:57.440480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:57.440482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:57.440541 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.440542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.441537 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.441538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.443019 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.443021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.443675 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.443676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.443737 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.443741 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:57.443789 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:52:57.443827 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":40,"threads_started":1}
I20260927 02:52:57.444092 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.440113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.0.98.1:43539)
0927 02:52:57.440262 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.440267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.440315 (+ 48us) server_negotiation.cc:407] Connection header received
0927 02:52:57.440381 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.440383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.440396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.440460 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:57.440715 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.441840 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.442704 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.443443 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.443515 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.443571 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.443960 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.444003 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260927 02:52:57.444151 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.439957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.0.98.3:32967)
0927 02:52:57.440379 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.440384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.440392 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:57.440424 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.440426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.440440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.440647 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:57.440705 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.441799 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.443009 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.443650 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.443950 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.444013 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.444037 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.444076 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":63,"threads_started":1}
I20260927 02:52:57.444332 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.440092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.0.98.2:34177)
0927 02:52:57.440263 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.440267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.440278 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:52:57.440339 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.440342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.440357 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.440459 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:57.440609 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.441452 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.443070 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.443585 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.443653 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.443707 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.443799 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.444262 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260927 02:52:57.444661 4580 tablet_service.cc:1467] Processing CreateTablet for tablet b0f3b21099a143f0aa38a79ad4a7afeb (DEFAULT_TABLE table=test-table [id=d66b2df281aa45608042d4f361fd1189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:57.444706 4846 tablet_service.cc:1467] Processing CreateTablet for tablet b0f3b21099a143f0aa38a79ad4a7afeb (DEFAULT_TABLE table=test-table [id=d66b2df281aa45608042d4f361fd1189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:57.444996 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f3b21099a143f0aa38a79ad4a7afeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:57.444998 4846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f3b21099a143f0aa38a79ad4a7afeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:57.445019 4713 tablet_service.cc:1467] Processing CreateTablet for tablet b0f3b21099a143f0aa38a79ad4a7afeb (DEFAULT_TABLE table=test-table [id=d66b2df281aa45608042d4f361fd1189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:57.445271 4713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f3b21099a143f0aa38a79ad4a7afeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:57.447535 4926 tablet_bootstrap.cc:492] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Bootstrap starting.
I20260927 02:52:57.448164 4928 tablet_bootstrap.cc:492] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Bootstrap starting.
I20260927 02:52:57.448369 4927 tablet_bootstrap.cc:492] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41: Bootstrap starting.
I20260927 02:52:57.448515 4926 tablet_bootstrap.cc:654] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:57.448736 4928 tablet_bootstrap.cc:654] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:57.448894 4927 tablet_bootstrap.cc:654] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:57.449199 4926 log.cc:824] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:57.449240 4928 log.cc:824] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:57.449352 4927 log.cc:824] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:57.449898 4928 tablet_bootstrap.cc:492] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: No bootstrap required, opened a new log
I20260927 02:52:57.449906 4926 tablet_bootstrap.cc:492] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: No bootstrap required, opened a new log
I20260927 02:52:57.449965 4926 ts_tablet_manager.cc:1397] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:52:57.449965 4928 ts_tablet_manager.cc:1397] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:52:57.449996 4927 tablet_bootstrap.cc:492] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41: No bootstrap required, opened a new log
I20260927 02:52:57.450048 4927 ts_tablet_manager.cc:1397] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:52:57.451508 4927 raft_consensus.cc:348] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.451646 4927 raft_consensus.cc:374] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:57.451668 4927 raft_consensus.cc:729] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c77e54a9074de6bbd27e4867babc41, State: Initialized, Role: FOLLOWER
I20260927 02:52:57.451751 4927 consensus_queue.cc:260] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.451845 4926 raft_consensus.cc:348] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.451843 4928 raft_consensus.cc:348] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.451984 4926 raft_consensus.cc:374] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:57.451984 4928 raft_consensus.cc:374] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:57.451982 4646 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
I20260927 02:52:57.452021 4928 raft_consensus.cc:729] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e14f7d83be41cb9d4e45a4714e3c96, State: Initialized, Role: FOLLOWER
I20260927 02:52:57.451989 4927 ts_tablet_manager.cc:1428] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:52:57.452021 4926 raft_consensus.cc:729] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a3d26dc0ad47ce912f86fa53414734, State: Initialized, Role: FOLLOWER
I20260927 02:52:57.452143 4928 consensus_queue.cc:260] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.452143 4926 consensus_queue.cc:260] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.452404 4928 ts_tablet_manager.cc:1428] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:52:57.452404 4912 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
I20260927 02:52:57.452504 4779 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
I20260927 02:52:57.452404 4926 ts_tablet_manager.cc:1428] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260927 02:52:57.523833 4780 tablet.cc:2369] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:52:57.626858 4647 tablet.cc:2369] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:52:57.666055 4933 raft_consensus.cc:482] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:57.666175 4933 raft_consensus.cc:504] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.666501 4933 leader_election.cc:288] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a3d26dc0ad47ce912f86fa53414734 (127.0.98.3:32967), 95c77e54a9074de6bbd27e4867babc41 (127.0.98.1:43539)
W20260927 02:52:57.666900 4913 tablet.cc:2369] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:52:57.669044 4781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.666658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43539 (local address 127.0.98.2:54927)
0927 02:52:57.666709 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.666720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:57.666792 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:57.666952 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:57.666953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:57.666954 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:57.667023 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.667024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.667620 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.667621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.668336 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.668338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.668828 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.668829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.668895 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.668898 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:57.668940 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:52:57.668978 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:52:57.669070 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.666703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54927 (local address 127.0.98.1:43539)
0927 02:52:57.666833 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.666835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.666845 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:52:57.666871 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.666872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.666884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.666946 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:57.667090 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.667546 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.668416 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.668759 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.668814 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.668824 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.668980 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.669015 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 02:52:57.669335 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.666732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42699 (local address 127.0.98.3:32967)
0927 02:52:57.667152 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.667154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.667161 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:52:57.667183 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.667184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.667195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.667251 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:52:57.667370 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.667808 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.668671 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.669011 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.669066 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.669076 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.669248 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.669279 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260927 02:52:57.669346 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.666666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32967 (local address 127.0.98.2:42699)
0927 02:52:57.667084 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.667093 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:57.667136 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:57.667258 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:57.667259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:57.667261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:52:57.667318 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.667319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.667880 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.667881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.668596 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:57.668597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.669098 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.669099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:57.669181 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.669185 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:52:57.669231 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:52:57.669270 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":12,"thread_start_us":328,"threads_started":1}
I20260927 02:52:57.669376 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f3b21099a143f0aa38a79ad4a7afeb" candidate_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c77e54a9074de6bbd27e4867babc41" is_pre_election: true
I20260927 02:52:57.669538 4600 raft_consensus.cc:2454] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e14f7d83be41cb9d4e45a4714e3c96 in term 0.
I20260927 02:52:57.669607 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f3b21099a143f0aa38a79ad4a7afeb" candidate_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a3d26dc0ad47ce912f86fa53414734" is_pre_election: true
I20260927 02:52:57.669756 4866 raft_consensus.cc:2454] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e14f7d83be41cb9d4e45a4714e3c96 in term 0.
I20260927 02:52:57.669772 4666 leader_election.cc:302] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95c77e54a9074de6bbd27e4867babc41, c7e14f7d83be41cb9d4e45a4714e3c96; no voters:
I20260927 02:52:57.669940 4933 raft_consensus.cc:2781] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:57.669984 4933 raft_consensus.cc:482] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:57.670004 4933 raft_consensus.cc:3037] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:57.670653 4933 raft_consensus.cc:504] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.670799 4933 leader_election.cc:288] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [CANDIDATE]: Term 1 election: Requested vote from peers 90a3d26dc0ad47ce912f86fa53414734 (127.0.98.3:32967), 95c77e54a9074de6bbd27e4867babc41 (127.0.98.1:43539)
I20260927 02:52:57.670914 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f3b21099a143f0aa38a79ad4a7afeb" candidate_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a3d26dc0ad47ce912f86fa53414734"
I20260927 02:52:57.670928 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f3b21099a143f0aa38a79ad4a7afeb" candidate_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c77e54a9074de6bbd27e4867babc41"
I20260927 02:52:57.670989 4866 raft_consensus.cc:3037] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:57.670989 4600 raft_consensus.cc:3037] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:57.671607 4600 raft_consensus.cc:2454] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e14f7d83be41cb9d4e45a4714e3c96 in term 1.
I20260927 02:52:57.671620 4866 raft_consensus.cc:2454] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e14f7d83be41cb9d4e45a4714e3c96 in term 1.
I20260927 02:52:57.671746 4668 leader_election.cc:302] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a3d26dc0ad47ce912f86fa53414734, c7e14f7d83be41cb9d4e45a4714e3c96; no voters:
I20260927 02:52:57.671841 4933 raft_consensus.cc:2781] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:57.671972 4933 raft_consensus.cc:686] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 1 LEADER]: Becoming Leader. State: Replica: c7e14f7d83be41cb9d4e45a4714e3c96, State: Running, Role: LEADER
I20260927 02:52:57.672060 4933 consensus_queue.cc:237] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } }
I20260927 02:52:57.672842 4456 catalog_manager.cc:5495] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7e14f7d83be41cb9d4e45a4714e3c96 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e14f7d83be41cb9d4e45a4714e3c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 } health_report { overall_health: UNKNOWN } } }
I20260927 02:52:57.724148 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.721986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.2:34177)
0927 02:52:57.722049 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.722050 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.722058 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:57.722090 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.722091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.722108 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.722165 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:57.722274 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.722721 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.723470 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:57.723645 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:57.723690 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:57.723884 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:57.723984 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:57.724036 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.724065 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.724080 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:52:57.725843 4866 raft_consensus.cc:1261] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 1 FOLLOWER]: Refusing update from remote peer c7e14f7d83be41cb9d4e45a4714e3c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:57.725914 4600 raft_consensus.cc:1261] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 1 FOLLOWER]: Refusing update from remote peer c7e14f7d83be41cb9d4e45a4714e3c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:57.726171 4936 consensus_queue.cc:1035] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a3d26dc0ad47ce912f86fa53414734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:57.726275 4933 consensus_queue.cc:1035] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95c77e54a9074de6bbd27e4867babc41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:57.728825 4941 mvcc.cc:205] Tried to move back new op lower bound from 7333796158362476544 to 7333796158145142784. Current Snapshot: MvccSnapshot[applied={T|T < 7333796158362476544}]
I20260927 02:52:57.730997 4456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:57.731083 4456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:57.731437 4940 mvcc.cc:205] Tried to move back new op lower bound from 7333796158362476544 to 7333796158145142784. Current Snapshot: MvccSnapshot[applied={T|T < 7333796158362476544 or (T in {7333796158362476544})}]
I20260927 02:52:57.731799 4456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65c517fe9d73442f9d30b398c370de40: Sending DeleteTablet for 3 replicas of tablet b0f3b21099a143f0aa38a79ad4a7afeb
I20260927 02:52:57.732123 4580 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f3b21099a143f0aa38a79ad4a7afeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60424
I20260927 02:52:57.732139 4713 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f3b21099a143f0aa38a79ad4a7afeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58200
I20260927 02:52:57.732146 4846 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f3b21099a143f0aa38a79ad4a7afeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54836
I20260927 02:52:57.732384 4949 tablet_replica.cc:330] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41: stopping tablet replica
I20260927 02:52:57.732549 4949 raft_consensus.cc:2229] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:57.732646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 4518
I20260927 02:52:57.732671 4949 raft_consensus.cc:2258] T b0f3b21099a143f0aa38a79ad4a7afeb P 95c77e54a9074de6bbd27e4867babc41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:57.733915 4951 tablet_replica.cc:330] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: stopping tablet replica
I20260927 02:52:57.734015 4951 raft_consensus.cc:2229] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:52:57.734108 4951 raft_consensus.cc:2258] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:57.734256 4950 tablet_replica.cc:330] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: stopping tablet replica
I20260927 02:52:57.734396 4950 raft_consensus.cc:2229] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:57.734544 4950 raft_consensus.cc:2258] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:57.734597 4951 ts_tablet_manager.cc:1905] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:57.735023 4950 ts_tablet_manager.cc:1905] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:57.736363 4951 ts_tablet_manager.cc:1918] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:52:57.736506 4951 log.cc:1196] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/b0f3b21099a143f0aa38a79ad4a7afeb
I20260927 02:52:57.736538 4950 ts_tablet_manager.cc:1918] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:52:57.736614 4950 log.cc:1196] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/b0f3b21099a143f0aa38a79ad4a7afeb
I20260927 02:52:57.736924 4950 ts_tablet_manager.cc:1939] T b0f3b21099a143f0aa38a79ad4a7afeb P 90a3d26dc0ad47ce912f86fa53414734: Deleting consensus metadata
I20260927 02:52:57.736924 4951 ts_tablet_manager.cc:1939] T b0f3b21099a143f0aa38a79ad4a7afeb P c7e14f7d83be41cb9d4e45a4714e3c96: Deleting consensus metadata
I20260927 02:52:57.737360 4442 catalog_manager.cc:4841] TS 90a3d26dc0ad47ce912f86fa53414734 (127.0.98.3:32967): tablet b0f3b21099a143f0aa38a79ad4a7afeb (table test-table [id=d66b2df281aa45608042d4f361fd1189]) successfully deleted
I20260927 02:52:57.737412 4441 catalog_manager.cc:4841] TS c7e14f7d83be41cb9d4e45a4714e3c96 (127.0.98.2:34177): tablet b0f3b21099a143f0aa38a79ad4a7afeb (table test-table [id=d66b2df281aa45608042d4f361fd1189]) successfully deleted
W20260927 02:52:57.737421 4440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43539 (error 108)
I20260927 02:52:57.737748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 4651
I20260927 02:52:57.738516 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.737909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43539 (local address 127.0.0.1:60432)
0927 02:52:57.738441 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:57.738483 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511}
W20260927 02:52:57.738641 4440 catalog_manager.cc:4584] TS 95c77e54a9074de6bbd27e4867babc41 (127.0.98.1:43539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0f3b21099a143f0aa38a79ad4a7afeb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43539: connect: Connection refused (error 111)
I20260927 02:52:57.742146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 4784
I20260927 02:52:57.746187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 4425
2026-09-27T02:52:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1961 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T02:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:57Z Disabled control of system clock
I20260927 02:52:57.770072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37547
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:52:57.846848 4956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:57.847023 4956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:57.847041 4956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:57.848093 4956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:57.848122 4956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:57.848133 4956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:57.848143 4956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:57.848157 4956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:52:57.849398 4956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:57.849570 4956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:57.849792 4956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:57.854067 4961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:57.854251 4956 server_base.cc:1034] running on GCE node
W20260927 02:52:57.854077 4964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:57.854094 4962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:57.854558 4956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:57.854799 4956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:57.855931 4956 hybrid_clock.cc:648] HybridClock initialized: now 1790477577855917 us; error 28 us; skew 500 ppm
I20260927 02:52:57.856986 4956 webserver.cc:460] Webserver started at http://127.0.98.62:38823/ using document root <none> and password file <none>
I20260927 02:52:57.857187 4956 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:57.857227 4956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:57.857342 4956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:57.858296 4956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "d45c3b62e5fc418e8fa507e00cc7a918"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "381b2657dae20a5ed9f8a23f4ed6c589"
server_key_iv: "0d5d0c99d31033597176766c644d3c57"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.858600 4956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "d45c3b62e5fc418e8fa507e00cc7a918"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "381b2657dae20a5ed9f8a23f4ed6c589"
server_key_iv: "0d5d0c99d31033597176766c644d3c57"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.859714 4956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:52:57.860340 4970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.860463 4956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:57.860536 4956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "d45c3b62e5fc418e8fa507e00cc7a918"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "381b2657dae20a5ed9f8a23f4ed6c589"
server_key_iv: "0d5d0c99d31033597176766c644d3c57"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.860599 4956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:57.863807 4956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:57.864023 4956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:57.864128 4956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:57.867520 4956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37547
I20260927 02:52:57.867548 5022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37547 every 8 connection(s)
I20260927 02:52:57.867833 4956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:57.868424 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:57.871040 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918: Bootstrap starting.
I20260927 02:52:57.871520 5023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:57.871748 5023 log.cc:824] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:57.872282 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918: No bootstrap required, opened a new log
I20260927 02:52:57.873576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 4956
I20260927 02:52:57.873646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:57.873790 392 external_mini_cluster.cc:1428] Setting key 12310c7df0c82074f3d2881564fcefa3
I20260927 02:52:57.873836 5023 raft_consensus.cc:348] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c3b62e5fc418e8fa507e00cc7a918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } }
I20260927 02:52:57.873989 5023 raft_consensus.cc:374] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:57.874030 5023 raft_consensus.cc:729] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45c3b62e5fc418e8fa507e00cc7a918, State: Initialized, Role: FOLLOWER
I20260927 02:52:57.874168 5023 consensus_queue.cc:260] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c3b62e5fc418e8fa507e00cc7a918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } }
I20260927 02:52:57.874294 5023 raft_consensus.cc:388] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:57.874370 5023 raft_consensus.cc:482] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:57.874426 5023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:57.875077 5023 raft_consensus.cc:504] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c3b62e5fc418e8fa507e00cc7a918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } }
I20260927 02:52:57.875211 5023 leader_election.cc:302] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d45c3b62e5fc418e8fa507e00cc7a918; no voters:
I20260927 02:52:57.875367 5023 leader_election.cc:288] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:57.875399 5028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:57.875555 5028 raft_consensus.cc:686] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [term 1 LEADER]: Becoming Leader. State: Replica: d45c3b62e5fc418e8fa507e00cc7a918, State: Running, Role: LEADER
I20260927 02:52:57.875600 5023 sys_catalog.cc:564] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:57.875679 5028 consensus_queue.cc:237] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c3b62e5fc418e8fa507e00cc7a918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } }
I20260927 02:52:57.875976 5030 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d45c3b62e5fc418e8fa507e00cc7a918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c3b62e5fc418e8fa507e00cc7a918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } } }
I20260927 02:52:57.876057 5030 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:57.876219 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:57.874223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.0.98.62:37547)
0927 02:52:57.874432 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:57.874442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:57.874473 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:52:57.874580 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:57.874587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:57.874628 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:57.874823 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:57.875391 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:57.875395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:57.875395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:57.875400 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:57.875445 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:57.875461 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:57.875523 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:57.875663 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:52:57.875720 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260927 02:52:57.875967 5028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45c3b62e5fc418e8fa507e00cc7a918. Latest consensus state: current_term: 1 leader_uuid: "d45c3b62e5fc418e8fa507e00cc7a918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c3b62e5fc418e8fa507e00cc7a918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } } }
I20260927 02:52:57.876868 5028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918 [sys.catalog]: This master's current role is: LEADER
W20260927 02:52:57.877285 5044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:52:57.877326 5044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:52:57.877442 5034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:57.877581 5034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:57.878827 5034 catalog_manager.cc:1347] Generated new cluster ID: 6c2f1a4d726546159db58339f26fc49c
I20260927 02:52:57.878878 5034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:57.889314 5034 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:57.889464 5034 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:57.894773 5034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918: Generated new TSK 0
I20260927 02:52:57.894934 5034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:52:57.896674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37547
--builtin_ntp_servers=127.0.98.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:52:57.973973 5047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:57.974167 5047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:57.974196 5047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:57.975358 5047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:57.975394 5047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:57.975426 5047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:52:57.976892 5047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:57.977103 5047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:57.977357 5047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:57.982019 5053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:57.982020 5052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:57.982323 5055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:57.982353 5047 server_base.cc:1034] running on GCE node
I20260927 02:52:57.982583 5047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:57.982846 5047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:57.983999 5047 hybrid_clock.cc:648] HybridClock initialized: now 1790477577983981 us; error 30 us; skew 500 ppm
I20260927 02:52:57.985458 5047 webserver.cc:460] Webserver started at http://127.0.98.1:34471/ using document root <none> and password file <none>
I20260927 02:52:57.985706 5047 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:57.985764 5047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:57.985898 5047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:57.987303 5047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "8301b92f8e3d48d8a9a969eb8e4dd848"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "2d86a59fa6774e87219103c214078c05"
server_key_iv: "16623b94b035624f849be6557f33f338"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.987699 5047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8301b92f8e3d48d8a9a969eb8e4dd848"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "2d86a59fa6774e87219103c214078c05"
server_key_iv: "16623b94b035624f849be6557f33f338"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.989239 5047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:52:57.990089 5061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.990242 5047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:57.990342 5047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "8301b92f8e3d48d8a9a969eb8e4dd848"
format_stamp: "Formatted at 2026-09-27 02:52:57 on dist-test-slave-mfx9"
server_key: "2d86a59fa6774e87219103c214078c05"
server_key_iv: "16623b94b035624f849be6557f33f338"
server_key_version: "encryptionkey@0"
I20260927 02:52:57.990415 5047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:57.994772 5047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:57.995033 5047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:57.995142 5047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:57.995323 5047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:57.995615 5047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:57.995651 5047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:57.995674 5047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:57.995697 5047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.002077 5047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37531
I20260927 02:52:58.002140 5174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37531 every 8 connection(s)
I20260927 02:52:58.002444 5047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:52:58.005311 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.003209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57543 (local address 127.0.98.62:37547)
0927 02:52:58.003270 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.003272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.003490 (+ 218us) server_negotiation.cc:407] Connection header received
0927 02:52:58.003966 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.003967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.003973 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.004024 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.005139 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.005141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.005141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.005144 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.005157 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.005161 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.005204 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.005250 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.005261 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:52:58.005512 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.003190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37547 (local address 127.0.98.1:57543)
0927 02:52:58.003373 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.003407 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.003888 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.004082 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.004086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.004094 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:58.004999 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:52:58.005000 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:58.005030 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:52:58.005058 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:58.005203 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:58.005205 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:58.005208 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:58.005257 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:52:58.005304 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260927 02:52:58.005982 5175 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37547
I20260927 02:52:58.006064 5175 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:58.006194 5175 heartbeater.cc:511] Master 127.0.98.62:37547 requested a full tablet report, sending...
I20260927 02:52:58.006618 4987 ts_manager.cc:194] Registered new tserver with Master: 8301b92f8e3d48d8a9a969eb8e4dd848 (127.0.98.1:37531)
I20260927 02:52:58.011080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 5047
I20260927 02:52:58.011173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:52:58.011305 392 external_mini_cluster.cc:1428] Setting key 07ac8fb58c5d64ad0bbb29e83e2da62f
I20260927 02:52:58.012149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37547
--builtin_ntp_servers=127.0.98.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:52:58.100852 5178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:58.101080 5178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:58.101112 5178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:58.102495 5178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:58.102591 5178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:58.102646 5178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:52:58.104350 5178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:58.104557 5178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:58.104796 5178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:58.109436 5183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:58.109512 5184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:58.109441 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:58.109658 5178 server_base.cc:1034] running on GCE node
I20260927 02:52:58.109825 5178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:58.110055 5178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:58.111191 5178 hybrid_clock.cc:648] HybridClock initialized: now 1790477578111180 us; error 26 us; skew 500 ppm
I20260927 02:52:58.112401 5178 webserver.cc:460] Webserver started at http://127.0.98.2:37705/ using document root <none> and password file <none>
I20260927 02:52:58.112613 5178 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:58.112670 5178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:58.112794 5178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:58.113838 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "02784945d3a2461ab0033a9fc65d9d41"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "3226a44cae4948f74583d8d72e2e973c"
server_key_iv: "431420fa3228b1359a01310e6ef170a9"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.114168 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "02784945d3a2461ab0033a9fc65d9d41"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "3226a44cae4948f74583d8d72e2e973c"
server_key_iv: "431420fa3228b1359a01310e6ef170a9"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.115396 5178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:58.115998 5192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.116127 5178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:58.116199 5178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "02784945d3a2461ab0033a9fc65d9d41"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "3226a44cae4948f74583d8d72e2e973c"
server_key_iv: "431420fa3228b1359a01310e6ef170a9"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.116268 5178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:58.119481 5178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:58.119700 5178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:58.119808 5178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:58.119980 5178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:58.120252 5178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:58.120282 5178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.120301 5178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:58.120323 5178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.125425 5178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40685
I20260927 02:52:58.125476 5305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40685 every 8 connection(s)
I20260927 02:52:58.125741 5178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:52:58.126256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 5178
I20260927 02:52:58.126359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:52:58.126538 392 external_mini_cluster.cc:1428] Setting key 180c8e66846362dd6fa9f2fd0404bd16
I20260927 02:52:58.127449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37547
--builtin_ntp_servers=127.0.98.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 02:52:58.128592 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.126371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52835 (local address 127.0.98.62:37547)
0927 02:52:58.126415 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.126417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.126660 (+ 243us) server_negotiation.cc:407] Connection header received
0927 02:52:58.127231 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.127232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.127238 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.127276 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.128366 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.128367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.128368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.128372 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.128389 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.128394 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.128443 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.128512 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.128525 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:52:58.128799 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.126376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37547 (local address 127.0.98.2:52835)
0927 02:52:58.126550 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.126584 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.127126 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.127340 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.127343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.127353 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:58.128228 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:52:58.128228 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:58.128255 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:52:58.128283 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:58.128449 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:58.128454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:58.128457 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:58.128523 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:52:58.128580 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260927 02:52:58.129189 5306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37547
I20260927 02:52:58.129266 5306 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:58.129369 5306 heartbeater.cc:511] Master 127.0.98.62:37547 requested a full tablet report, sending...
I20260927 02:52:58.129684 4987 ts_manager.cc:194] Registered new tserver with Master: 02784945d3a2461ab0033a9fc65d9d41 (127.0.98.2:40685)
W20260927 02:52:58.206234 5309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:58.206431 5309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:58.206451 5309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:58.207542 5309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:58.207576 5309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:58.207598 5309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:52:58.208894 5309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:58.209074 5309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:58.209306 5309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:58.214008 5314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:58.214037 5315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:58.214047 5309 server_base.cc:1034] running on GCE node
W20260927 02:52:58.214008 5317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:58.214414 5309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:58.214632 5309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:58.215762 5309 hybrid_clock.cc:648] HybridClock initialized: now 1790477578215752 us; error 26 us; skew 500 ppm
I20260927 02:52:58.216926 5309 webserver.cc:460] Webserver started at http://127.0.98.3:41905/ using document root <none> and password file <none>
I20260927 02:52:58.217130 5309 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:58.217177 5309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:58.217312 5309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:58.218326 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "de186f0033534f04ac5dbe5ed7e0f2a1"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "fb892a536019c5b01ef7e9856e99edd7"
server_key_iv: "1667c3e375248e34d0a8351e3ec1a23d"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.218685 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "de186f0033534f04ac5dbe5ed7e0f2a1"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "fb892a536019c5b01ef7e9856e99edd7"
server_key_iv: "1667c3e375248e34d0a8351e3ec1a23d"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.219908 5309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:52:58.220532 5323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.220676 5309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:58.220753 5309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "de186f0033534f04ac5dbe5ed7e0f2a1"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "fb892a536019c5b01ef7e9856e99edd7"
server_key_iv: "1667c3e375248e34d0a8351e3ec1a23d"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.220817 5309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:58.224030 5309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:58.224295 5309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:58.224418 5309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:58.224612 5309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:52:58.224893 5309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:52:58.224925 5309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.224951 5309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:52:58.224975 5309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.230269 5309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34261
I20260927 02:52:58.230304 5436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34261 every 8 connection(s)
I20260927 02:52:58.230602 5309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:52:58.231724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 5309
I20260927 02:52:58.231802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:52:58.231952 392 external_mini_cluster.cc:1428] Setting key d1a300794a33ef9a34ddc3af44b3c7fd
I20260927 02:52:58.233548 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.231323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46537 (local address 127.0.98.62:37547)
0927 02:52:58.231382 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.231385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.231593 (+ 208us) server_negotiation.cc:407] Connection header received
0927 02:52:58.232082 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.232083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.232089 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.232149 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.233312 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.233314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.233314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.233319 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.233334 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.233338 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.233386 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.233465 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.233480 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:52:58.233726 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.231316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37547 (local address 127.0.98.3:46537)
0927 02:52:58.231479 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.231512 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.231981 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.232221 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.232225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.232243 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:58.233162 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:52:58.233162 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:58.233191 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:52:58.233219 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:58.233394 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:58.233398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:58.233401 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:58.233474 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:52:58.233534 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260927 02:52:58.234064 5437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37547
I20260927 02:52:58.234144 5437 heartbeater.cc:464] Registering TS with master...
I20260927 02:52:58.234283 5437 heartbeater.cc:511] Master 127.0.98.62:37547 requested a full tablet report, sending...
I20260927 02:52:58.234565 4987 ts_manager.cc:194] Registered new tserver with Master: de186f0033534f04ac5dbe5ed7e0f2a1 (127.0.98.3:34261)
I20260927 02:52:58.243597 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:52:58.245738 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.244791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.0.98.62:37547)
0927 02:52:58.244872 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.244874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.244970 (+ 96us) server_negotiation.cc:407] Connection header received
0927 02:52:58.245021 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.245021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.245026 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.245062 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.245556 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.245557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.245557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.245561 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.245576 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.245580 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.245627 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.245659 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.245671 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:52:58.247140 4987 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:52:58.248565 4987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:52:58.251778 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.250415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37531 (local address 127.0.0.1:50548)
0927 02:52:58.250689 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.250696 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.250724 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.250969 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.250974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.250980 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:58.251406 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:52:58.251406 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:58.251422 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:52:58.251454 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:58.251640 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:58.251645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:58.251648 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:52:58.251703 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:52:58.251715 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":89,"threads_started":1}
I20260927 02:52:58.251957 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.250630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.0.98.1:37531)
0927 02:52:58.250851 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.250855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.250863 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:58.250883 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.250885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.250892 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.250953 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.251537 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.251542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.251542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.251546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.251574 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.251589 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.251637 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.251875 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.251891 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260927 02:52:58.252264 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.250374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40685 (local address 127.0.0.1:41708)
0927 02:52:58.250535 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.250563 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.250615 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.251007 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.251009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.251012 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:58.251738 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:52:58.251739 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:58.251744 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:52:58.251752 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:58.252167 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:58.252168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:58.252168 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 02:52:58.252209 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:52:58.252219 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260927 02:52:58.252317 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.250656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.0.98.2:40685)
0927 02:52:58.250847 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.250851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.250862 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:52:58.250887 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.250890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.250899 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.251815 (+ 916us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.252065 (+ 250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.252068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.252068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.252071 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.252097 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.252108 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.252164 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.252216 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.252239 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260927 02:52:58.252764 5109 tablet_service.cc:1467] Processing CreateTablet for tablet 85df7610d1944831bf531e0b90612ef8 (DEFAULT_TABLE table=test-table [id=19ac634a75a140cea79cf5fb5afcd124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:58.252888 5240 tablet_service.cc:1467] Processing CreateTablet for tablet 85df7610d1944831bf531e0b90612ef8 (DEFAULT_TABLE table=test-table [id=19ac634a75a140cea79cf5fb5afcd124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:58.253126 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85df7610d1944831bf531e0b90612ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:58.253149 5240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85df7610d1944831bf531e0b90612ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:58.254958 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.250374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34261 (local address 127.0.0.1:49198)
0927 02:52:58.250515 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.250563 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.250615 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.251308 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.251309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.251312 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:58.251627 (+ 315us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:52:58.251628 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:58.251635 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:52:58.251648 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:58.254831 (+ 3183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:58.254832 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:58.254833 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:52:58.254886 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:52:58.254899 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":45,"threads_started":1}
I20260927 02:52:58.254995 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.250513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.0.98.3:34261)
0927 02:52:58.250930 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.250934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.250942 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:52:58.250963 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.250965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.250972 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.251032 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.251717 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.251724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.251725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.251728 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.251771 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.251792 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.251847 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.254906 (+ 3059us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.254923 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":64,"threads_started":1}
I20260927 02:52:58.256286 5451 tablet_bootstrap.cc:492] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: Bootstrap starting.
I20260927 02:52:58.256698 5371 tablet_service.cc:1467] Processing CreateTablet for tablet 85df7610d1944831bf531e0b90612ef8 (DEFAULT_TABLE table=test-table [id=19ac634a75a140cea79cf5fb5afcd124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:52:58.256971 5371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85df7610d1944831bf531e0b90612ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:58.257009 5451 tablet_bootstrap.cc:654] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:58.257278 5451 log.cc:824] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:58.257601 5452 tablet_bootstrap.cc:492] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848: Bootstrap starting.
I20260927 02:52:58.257999 5451 tablet_bootstrap.cc:492] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: No bootstrap required, opened a new log
I20260927 02:52:58.258129 5451 ts_tablet_manager.cc:1397] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:52:58.258257 5452 tablet_bootstrap.cc:654] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:58.258535 5452 log.cc:824] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:58.259172 5452 tablet_bootstrap.cc:492] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848: No bootstrap required, opened a new log
I20260927 02:52:58.259234 5452 ts_tablet_manager.cc:1397] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:52:58.259968 5455 tablet_bootstrap.cc:492] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: Bootstrap starting.
I20260927 02:52:58.260505 5451 raft_consensus.cc:348] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.260632 5451 raft_consensus.cc:374] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:58.260663 5451 raft_consensus.cc:729] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02784945d3a2461ab0033a9fc65d9d41, State: Initialized, Role: FOLLOWER
I20260927 02:52:58.260682 5455 tablet_bootstrap.cc:654] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:58.260815 5451 consensus_queue.cc:260] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.260980 5455 log.cc:824] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:58.261059 5451 ts_tablet_manager.cc:1428] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:52:58.261171 5306 heartbeater.cc:503] Master 127.0.98.62:37547 was elected leader, sending a full tablet report...
I20260927 02:52:58.261600 5452 raft_consensus.cc:348] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.261772 5452 raft_consensus.cc:374] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:58.261844 5452 raft_consensus.cc:729] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8301b92f8e3d48d8a9a969eb8e4dd848, State: Initialized, Role: FOLLOWER
I20260927 02:52:58.261968 5452 consensus_queue.cc:260] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.262205 5452 ts_tablet_manager.cc:1428] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:52:58.262297 5175 heartbeater.cc:503] Master 127.0.98.62:37547 was elected leader, sending a full tablet report...
I20260927 02:52:58.262821 5455 tablet_bootstrap.cc:492] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: No bootstrap required, opened a new log
I20260927 02:52:58.262897 5455 ts_tablet_manager.cc:1397] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:52:58.264686 5455 raft_consensus.cc:348] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.264848 5455 raft_consensus.cc:374] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:58.264895 5455 raft_consensus.cc:729] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de186f0033534f04ac5dbe5ed7e0f2a1, State: Initialized, Role: FOLLOWER
I20260927 02:52:58.265007 5455 consensus_queue.cc:260] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.265230 5455 ts_tablet_manager.cc:1428] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:52:58.265266 5437 heartbeater.cc:503] Master 127.0.98.62:37547 was elected leader, sending a full tablet report...
I20260927 02:52:58.290169 5456 raft_consensus.cc:482] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:52:58.290305 5456 raft_consensus.cc:504] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.290623 5456 leader_election.cc:288] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de186f0033534f04ac5dbe5ed7e0f2a1 (127.0.98.3:34261), 8301b92f8e3d48d8a9a969eb8e4dd848 (127.0.98.1:37531)
I20260927 02:52:58.291836 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.290929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39113 (local address 127.0.98.3:34261)
0927 02:52:58.290987 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.290989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.290998 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:52:58.291028 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.291029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.291035 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.291109 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.291638 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.291639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.291640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.291644 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.291659 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.291664 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.291713 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.291761 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.291772 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:52:58.291836 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.290837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34261 (local address 127.0.98.2:39113)
0927 02:52:58.290879 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.290890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.290949 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.291116 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.291117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.291119 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:58.291557 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:52:58.291557 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:58.291563 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:52:58.291573 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:58.291714 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:58.291715 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:58.291715 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 02:52:58.291757 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:52:58.291769 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:52:58.292181 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.290859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37531 (local address 127.0.98.2:53899)
0927 02:52:58.291041 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.291051 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.291101 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.291238 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.291239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.291241 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:52:58.291711 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:52:58.291711 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:52:58.291718 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:52:58.291732 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:52:58.292030 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:52:58.292030 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:52:58.292031 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:52:58.292082 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:52:58.292095 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260927 02:52:58.292228 5391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85df7610d1944831bf531e0b90612ef8" candidate_uuid: "02784945d3a2461ab0033a9fc65d9d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" is_pre_election: true
I20260927 02:52:58.292382 5391 raft_consensus.cc:2454] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02784945d3a2461ab0033a9fc65d9d41 in term 0.
I20260927 02:52:58.292588 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.290916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53899 (local address 127.0.98.1:37531)
0927 02:52:58.290969 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.290971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.291108 (+ 137us) server_negotiation.cc:407] Connection header received
0927 02:52:58.291161 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.291162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.291168 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.291221 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.291819 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.291820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.291820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.291825 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.291842 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.291847 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.291902 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.292516 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.292531 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:52:58.292600 5196 leader_election.cc:302] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02784945d3a2461ab0033a9fc65d9d41, de186f0033534f04ac5dbe5ed7e0f2a1; no voters:
I20260927 02:52:58.292716 5456 raft_consensus.cc:2781] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:52:58.292764 5456 raft_consensus.cc:482] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:52:58.292786 5456 raft_consensus.cc:3037] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:58.292860 5129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85df7610d1944831bf531e0b90612ef8" candidate_uuid: "02784945d3a2461ab0033a9fc65d9d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" is_pre_election: true
I20260927 02:52:58.292994 5129 raft_consensus.cc:2454] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02784945d3a2461ab0033a9fc65d9d41 in term 0.
I20260927 02:52:58.293587 5456 raft_consensus.cc:504] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.293700 5456 leader_election.cc:288] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [CANDIDATE]: Term 1 election: Requested vote from peers de186f0033534f04ac5dbe5ed7e0f2a1 (127.0.98.3:34261), 8301b92f8e3d48d8a9a969eb8e4dd848 (127.0.98.1:37531)
I20260927 02:52:58.293850 5129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85df7610d1944831bf531e0b90612ef8" candidate_uuid: "02784945d3a2461ab0033a9fc65d9d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848"
I20260927 02:52:58.293834 5391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85df7610d1944831bf531e0b90612ef8" candidate_uuid: "02784945d3a2461ab0033a9fc65d9d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1"
I20260927 02:52:58.293911 5391 raft_consensus.cc:3037] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:58.293921 5129 raft_consensus.cc:3037] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:58.294598 5129 raft_consensus.cc:2454] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02784945d3a2461ab0033a9fc65d9d41 in term 1.
I20260927 02:52:58.294687 5391 raft_consensus.cc:2454] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02784945d3a2461ab0033a9fc65d9d41 in term 1.
I20260927 02:52:58.294806 5195 leader_election.cc:302] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02784945d3a2461ab0033a9fc65d9d41, 8301b92f8e3d48d8a9a969eb8e4dd848; no voters:
I20260927 02:52:58.294884 5456 raft_consensus.cc:2781] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:58.295058 5456 raft_consensus.cc:686] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 1 LEADER]: Becoming Leader. State: Replica: 02784945d3a2461ab0033a9fc65d9d41, State: Running, Role: LEADER
I20260927 02:52:58.295153 5456 consensus_queue.cc:237] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } }
I20260927 02:52:58.295921 4987 catalog_manager.cc:5495] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02784945d3a2461ab0033a9fc65d9d41 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "02784945d3a2461ab0033a9fc65d9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02784945d3a2461ab0033a9fc65d9d41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 } health_report { overall_health: UNKNOWN } } }
I20260927 02:52:58.309363 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.308611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.0.98.2:40685)
0927 02:52:58.308645 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.308647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.308654 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:52:58.308709 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.308709 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.308719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.308771 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.309203 (+ 432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.309205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.309205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.309208 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.309220 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.309224 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.309275 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.309308 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.309317 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:52:58.310938 5129 raft_consensus.cc:1261] T 85df7610d1944831bf531e0b90612ef8 P 8301b92f8e3d48d8a9a969eb8e4dd848 [term 1 FOLLOWER]: Refusing update from remote peer 02784945d3a2461ab0033a9fc65d9d41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:58.310927 5391 raft_consensus.cc:1261] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 1 FOLLOWER]: Refusing update from remote peer 02784945d3a2461ab0033a9fc65d9d41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:52:58.311255 5456 consensus_queue.cc:1035] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8301b92f8e3d48d8a9a969eb8e4dd848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:58.311381 5461 consensus_queue.cc:1035] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de186f0033534f04ac5dbe5ed7e0f2a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:52:58.313452 5464 mvcc.cc:205] Tried to move back new op lower bound from 7333796160759152640 to 7333796160697634816. Current Snapshot: MvccSnapshot[applied={T|T < 7333796160759152640}]
I20260927 02:52:58.314307 5465 mvcc.cc:205] Tried to move back new op lower bound from 7333796160759152640 to 7333796160697634816. Current Snapshot: MvccSnapshot[applied={T|T < 7333796160759152640 or (T in {7333796160759152640})}]
I20260927 02:52:58.315585 4986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:58.315673 4986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:52:58.316314 4986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d45c3b62e5fc418e8fa507e00cc7a918: Sending DeleteTablet for 3 replicas of tablet 85df7610d1944831bf531e0b90612ef8
I20260927 02:52:58.316602 5371 tablet_service.cc:1514] Processing DeleteTablet for tablet 85df7610d1944831bf531e0b90612ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:58 UTC) from {username='slave'} at 127.0.0.1:49198
I20260927 02:52:58.316648 5109 tablet_service.cc:1514] Processing DeleteTablet for tablet 85df7610d1944831bf531e0b90612ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:58 UTC) from {username='slave'} at 127.0.0.1:50548
I20260927 02:52:58.316800 5240 tablet_service.cc:1514] Processing DeleteTablet for tablet 85df7610d1944831bf531e0b90612ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:52:58 UTC) from {username='slave'} at 127.0.0.1:41708
I20260927 02:52:58.316972 5476 tablet_replica.cc:330] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: stopping tablet replica
I20260927 02:52:58.317030 5474 tablet_replica.cc:330] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: stopping tablet replica
I20260927 02:52:58.317081 5476 raft_consensus.cc:2229] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:52:58.317178 5474 raft_consensus.cc:2229] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:52:58.317195 5476 raft_consensus.cc:2258] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:58.317237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 5047
I20260927 02:52:58.317937 5474 raft_consensus.cc:2258] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:52:58.318048 5476 ts_tablet_manager.cc:1905] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:58.318336 5474 ts_tablet_manager.cc:1905] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:52:58.319782 5474 ts_tablet_manager.cc:1918] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:52:58.319792 5476 ts_tablet_manager.cc:1918] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:52:58.319839 5474 log.cc:1196] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/85df7610d1944831bf531e0b90612ef8
I20260927 02:52:58.319849 5476 log.cc:1196] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/85df7610d1944831bf531e0b90612ef8
I20260927 02:52:58.320093 5476 ts_tablet_manager.cc:1939] T 85df7610d1944831bf531e0b90612ef8 P 02784945d3a2461ab0033a9fc65d9d41: Deleting consensus metadata
I20260927 02:52:58.320118 5474 ts_tablet_manager.cc:1939] T 85df7610d1944831bf531e0b90612ef8 P de186f0033534f04ac5dbe5ed7e0f2a1: Deleting consensus metadata
I20260927 02:52:58.320545 4974 catalog_manager.cc:4841] TS de186f0033534f04ac5dbe5ed7e0f2a1 (127.0.98.3:34261): tablet 85df7610d1944831bf531e0b90612ef8 (table test-table [id=19ac634a75a140cea79cf5fb5afcd124]) successfully deleted
I20260927 02:52:58.320655 4971 catalog_manager.cc:4841] TS 02784945d3a2461ab0033a9fc65d9d41 (127.0.98.2:40685): tablet 85df7610d1944831bf531e0b90612ef8 (table test-table [id=19ac634a75a140cea79cf5fb5afcd124]) successfully deleted
W20260927 02:52:58.323372 4973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37531 (error 108)
I20260927 02:52:58.323683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 5178
I20260927 02:52:58.323858 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.323745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37531 (local address 127.0.0.1:50558)
0927 02:52:58.323791 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.323828 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 02:52:58.324225 4973 catalog_manager.cc:4584] TS 8301b92f8e3d48d8a9a969eb8e4dd848 (127.0.98.1:37531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85df7610d1944831bf531e0b90612ef8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37531: connect: Connection refused (error 111)
I20260927 02:52:58.328367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 5309
I20260927 02:52:58.331751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 4956
2026-09-27T02:52:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (585 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 02:52:58.350158 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:52:58.351387 392 webserver.cc:460] Webserver started at https://127.0.0.1:36835/ using document root <none> and password file <none>
I20260927 02:52:58.351477 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:52:58.351646 392 webserver.cc:460] Webserver started at http://127.0.0.1:33653/ using document root <none> and password file <none>
2026-09-27T02:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:58Z Disabled control of system clock
I20260927 02:52:58.358805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43803
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43803
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36835/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/testchainca.cert with env {}
W20260927 02:52:58.437139 5483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:58.437340 5483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:58.437361 5483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:58.438457 5483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:58.438498 5483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:58.438512 5483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:58.438529 5483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:58.438545 5483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:52:58.438561 5483 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:52:58.438573 5483 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36835/jwks.json
I20260927 02:52:58.440037 5483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43803
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36835/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:58.440234 5483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:58.440399 5483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:58.444850 5491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:58.444898 5488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:58.444939 5483 server_base.cc:1034] running on GCE node
W20260927 02:52:58.444916 5489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:58.445308 5483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:58.445555 5483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:58.446686 5483 hybrid_clock.cc:648] HybridClock initialized: now 1790477578446672 us; error 28 us; skew 500 ppm
I20260927 02:52:58.447806 5483 webserver.cc:460] Webserver started at http://127.0.98.62:37495/ using document root <none> and password file <none>
I20260927 02:52:58.448004 5483 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:58.448050 5483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:58.448159 5483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:58.449126 5483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "c868e7f1cc394af3886761a94c111a08"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "af33d1b5ea8d0b6dd593c98e974e750b"
server_key_iv: "0e292bc0983ee33c3d15ce6fed25ee01"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.449430 5483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "c868e7f1cc394af3886761a94c111a08"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "af33d1b5ea8d0b6dd593c98e974e750b"
server_key_iv: "0e292bc0983ee33c3d15ce6fed25ee01"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.450498 5483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:58.451164 5497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.451303 5483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.451377 5483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "c868e7f1cc394af3886761a94c111a08"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "af33d1b5ea8d0b6dd593c98e974e750b"
server_key_iv: "0e292bc0983ee33c3d15ce6fed25ee01"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.451439 5483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:58.460002 5483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:58.460243 5483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:58.460369 5483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:58.464035 5483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43803
I20260927 02:52:58.464062 5549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43803 every 8 connection(s)
I20260927 02:52:58.464409 5483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:58.465219 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:58.467317 5550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08: Bootstrap starting.
I20260927 02:52:58.467857 5550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:58.468104 5550 log.cc:824] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:58.468662 5550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08: No bootstrap required, opened a new log
I20260927 02:52:58.470110 5550 raft_consensus.cc:348] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868e7f1cc394af3886761a94c111a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260927 02:52:58.470233 5550 raft_consensus.cc:374] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:58.470253 5550 raft_consensus.cc:729] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c868e7f1cc394af3886761a94c111a08, State: Initialized, Role: FOLLOWER
I20260927 02:52:58.470372 5550 consensus_queue.cc:260] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868e7f1cc394af3886761a94c111a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260927 02:52:58.470432 5550 raft_consensus.cc:388] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:58.470463 5550 raft_consensus.cc:482] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:58.470518 5550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:58.470996 5550 raft_consensus.cc:504] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868e7f1cc394af3886761a94c111a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260927 02:52:58.471077 5550 leader_election.cc:302] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c868e7f1cc394af3886761a94c111a08; no voters:
I20260927 02:52:58.471261 5550 leader_election.cc:288] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:58.471299 5553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:58.471433 5553 raft_consensus.cc:686] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [term 1 LEADER]: Becoming Leader. State: Replica: c868e7f1cc394af3886761a94c111a08, State: Running, Role: LEADER
I20260927 02:52:58.471486 5550 sys_catalog.cc:564] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:58.471524 5553 consensus_queue.cc:237] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868e7f1cc394af3886761a94c111a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260927 02:52:58.471982 5554 sys_catalog.cc:455] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c868e7f1cc394af3886761a94c111a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868e7f1cc394af3886761a94c111a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } } }
I20260927 02:52:58.472061 5554 sys_catalog.cc:458] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:58.472261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 5483
I20260927 02:52:58.472358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:58.472538 392 external_mini_cluster.cc:1428] Setting key 8519fb9fc0a72147ffb9e3a4bd645f21
I20260927 02:52:58.472844 5555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c868e7f1cc394af3886761a94c111a08. Latest consensus state: current_term: 1 leader_uuid: "c868e7f1cc394af3886761a94c111a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868e7f1cc394af3886761a94c111a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } } }
I20260927 02:52:58.472909 5555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08 [sys.catalog]: This master's current role is: LEADER
W20260927 02:52:58.473337 5568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:52:58.473387 5568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:52:58.473425 5570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:58.473573 5570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:58.474869 5570 catalog_manager.cc:1347] Generated new cluster ID: c00a5a6504d34a6aaf5d1a8d67b6b7ae
I20260927 02:52:58.474915 5570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:58.478277 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.473021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.0.98.62:43803)
0927 02:52:58.473376 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.473384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.473412 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:52:58.473528 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.473534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.473597 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.473722 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.473967 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.475285 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.476324 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.476939 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.477058 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.477598 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.477602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.477603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.477795 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.477834 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.477847 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.477919 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.477981 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.478055 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":60,"threads_started":1}
I20260927 02:52:58.479334 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:52:58.481361 5570 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:58.481832 5570 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:58.486653 5570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c868e7f1cc394af3886761a94c111a08: Generated new TSK 0
I20260927 02:52:58.486812 5570 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:52:58.516738 5580 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 02:52:58.516839 5580 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:52:58.516986 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.494455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.0.98.62:43803)
0927 02:52:58.494491 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.494492 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.494642 (+ 150us) server_negotiation.cc:407] Connection header received
0927 02:52:58.494726 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.494727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.494740 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.494870 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:52:58.495003 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.495460 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.496188 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.496378 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.496429 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.496605 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:52:58.516760 (+ 20155us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:52:58.516914 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 02:52:58.517056 5572 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 02:52:58.517432 5579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:52:58.494412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43803 (local address 127.0.0.1:55124)
0927 02:52:58.494582 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.494590 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.494652 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.494884 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.494885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.494887 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:52:58.494947 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:58.494948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.495540 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.495541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:58.496117 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:58.496118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.496432 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.496436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:58.496494 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.496515 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:52:58.516971 (+ 20456us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:52:58.517095 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43803: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43803: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 02:52:58.517956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 5483
2026-09-27T02:52:58Z chronyd exiting
I20260927 02:52:58.552973 392 test_util.cc:175] -----------------------------------------------
I20260927 02:52:58.553048 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477553417353-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (203 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 02:52:58.553753 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:52:58.554404 392 webserver.cc:460] Webserver started at https://127.0.0.1:41233/ using document root <none> and password file <none>
I20260927 02:52:58.554446 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:52:58.554631 392 webserver.cc:460] Webserver started at http://127.0.0.1:35961/ using document root <none> and password file <none>
2026-09-27T02:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:58Z Disabled control of system clock
I20260927 02:52:58.561405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33999
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33999
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41233/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/testchainca.cert with env {}
W20260927 02:52:58.639705 5586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:58.639896 5586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:58.639916 5586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:58.641023 5586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:58.641057 5586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:58.641067 5586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:58.641077 5586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:58.641094 5586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:52:58.641104 5586 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:52:58.641114 5586 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41233/jwks.json
I20260927 02:52:58.642514 5586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41233/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:58.642693 5586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:58.642899 5586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:58.647176 5594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:58.647197 5592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:58.647203 5591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:58.647265 5586 server_base.cc:1034] running on GCE node
I20260927 02:52:58.647591 5586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:58.647851 5586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:58.649003 5586 hybrid_clock.cc:648] HybridClock initialized: now 1790477578648985 us; error 27 us; skew 500 ppm
I20260927 02:52:58.650138 5586 webserver.cc:460] Webserver started at http://127.0.98.62:36571/ using document root <none> and password file <none>
I20260927 02:52:58.650341 5586 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:58.650388 5586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:58.650504 5586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:58.651499 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "014298f6a56348668eb284fecf7518e2"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "219066d9cf67a6b778c1f138fbc0ed1a"
server_key_iv: "93eb57a72f98732bd316d2c98429f7a9"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.651811 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "014298f6a56348668eb284fecf7518e2"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "219066d9cf67a6b778c1f138fbc0ed1a"
server_key_iv: "93eb57a72f98732bd316d2c98429f7a9"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.652936 5586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:52:58.653512 5600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.653653 5586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:58.653766 5586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "014298f6a56348668eb284fecf7518e2"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "219066d9cf67a6b778c1f138fbc0ed1a"
server_key_iv: "93eb57a72f98732bd316d2c98429f7a9"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.653831 5586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:58.670331 5586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:58.670581 5586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:58.670694 5586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:58.674180 5586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33999
I20260927 02:52:58.674209 5652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33999 every 8 connection(s)
I20260927 02:52:58.674484 5586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:58.675369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 5586
I20260927 02:52:58.675385 5653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:58.675477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:58.675661 392 external_mini_cluster.cc:1428] Setting key 0bba4cf3e54d8c9d52ebdb12d1eac730
I20260927 02:52:58.677997 5653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2: Bootstrap starting.
I20260927 02:52:58.678511 5653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:58.678742 5653 log.cc:824] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:58.679335 5653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2: No bootstrap required, opened a new log
I20260927 02:52:58.680464 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.676097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.0.98.62:33999)
0927 02:52:58.676290 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.676296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.676319 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:52:58.676737 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.676744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.676799 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.676915 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.677114 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.677957 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.678738 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.679199 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.679330 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.679915 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.679920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.679920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.680067 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.680097 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.680107 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.680155 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.680227 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.680307 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260927 02:52:58.681139 5653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "014298f6a56348668eb284fecf7518e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33999 } }
I20260927 02:52:58.681270 5653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:58.681290 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:52:58.681313 5653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014298f6a56348668eb284fecf7518e2, State: Initialized, Role: FOLLOWER
I20260927 02:52:58.681416 5653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "014298f6a56348668eb284fecf7518e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33999 } }
I20260927 02:52:58.681483 5653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:58.681505 5653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:58.681548 5653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:58.682070 5653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "014298f6a56348668eb284fecf7518e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33999 } }
I20260927 02:52:58.682158 5653 leader_election.cc:302] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 014298f6a56348668eb284fecf7518e2; no voters:
I20260927 02:52:58.682293 5653 leader_election.cc:288] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:58.682330 5658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:58.682447 5658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [term 1 LEADER]: Becoming Leader. State: Replica: 014298f6a56348668eb284fecf7518e2, State: Running, Role: LEADER
I20260927 02:52:58.682507 5653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:58.682524 5658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "014298f6a56348668eb284fecf7518e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33999 } }
I20260927 02:52:58.682857 5659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "014298f6a56348668eb284fecf7518e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "014298f6a56348668eb284fecf7518e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33999 } } }
I20260927 02:52:58.682858 5660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 014298f6a56348668eb284fecf7518e2. Latest consensus state: current_term: 1 leader_uuid: "014298f6a56348668eb284fecf7518e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "014298f6a56348668eb284fecf7518e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33999 } } }
I20260927 02:52:58.683000 5660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:58.682978 5659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:58.683393 5666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:58.683825 5666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:58.685143 5666 catalog_manager.cc:1347] Generated new cluster ID: 86a58cacf3ea406c8259a4afc7e04b4c
I20260927 02:52:58.685190 5666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:58.693655 5666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:58.694132 5666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:58.698153 5666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 014298f6a56348668eb284fecf7518e2: Generated new TSK 0
I20260927 02:52:58.698293 5666 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:52:58.724965 5683 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 02:52:58.725030 5683 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:52:58.725373 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.703917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.0.98.62:33999)
0927 02:52:58.703959 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.703962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.704062 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:52:58.704123 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.704124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.704135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.704190 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:52:58.704303 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.704763 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.705492 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.705675 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.705727 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.705867 (+ 140us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:52:58.725159 (+ 19292us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:52:58.725302 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54788: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 02:52:58.725422 5682 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:52:58.703868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33999 (local address 127.0.0.1:54788)
0927 02:52:58.704001 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.704011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.704068 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.704192 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.704193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.704194 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:52:58.704245 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:58.704247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.704845 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.704846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:58.705417 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:58.705418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.705730 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.705733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:58.705790 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.705795 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:52:58.725260 (+ 19465us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:52:58.725347 (+ 87us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33999: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260927 02:52:58.725430 5656 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54788: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41233/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 02:52:58.725651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 5586
2026-09-27T02:52:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (208 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 02:52:58.762328 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:52:58.763041 392 webserver.cc:460] Webserver started at https://127.0.0.1:38327/ using document root <none> and password file <none>
I20260927 02:52:58.763087 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:52:58.763212 392 webserver.cc:460] Webserver started at http://127.0.0.1:36561/ using document root <none> and password file <none>
2026-09-27T02:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:58Z Disabled control of system clock
I20260927 02:52:58.770210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42773
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42773
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38327/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:52:58.847836 5689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:58.848067 5689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:58.848096 5689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:58.849215 5689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:58.849253 5689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:58.849270 5689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:58.849287 5689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:58.849301 5689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:52:58.849318 5689 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:52:58.849339 5689 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38327/jwks.json
I20260927 02:52:58.850849 5689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38327/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:58.851083 5689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:58.851372 5689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:58.855651 5694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:58.855654 5695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:58.855674 5697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:58.855993 5689 server_base.cc:1034] running on GCE node
I20260927 02:52:58.856217 5689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:58.856452 5689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:58.857584 5689 hybrid_clock.cc:648] HybridClock initialized: now 1790477578857569 us; error 27 us; skew 500 ppm
I20260927 02:52:58.858698 5689 webserver.cc:460] Webserver started at http://127.0.98.62:33219/ using document root <none> and password file <none>
I20260927 02:52:58.858923 5689 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:58.858973 5689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:58.859092 5689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:58.860057 5689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "6f7ae15be8ad409aa8063cbfec61cd1e"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "8491c80d59cc387e563dedf39a881128"
server_key_iv: "19dfec293056572d98bed851e71dd87b"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.860368 5689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f7ae15be8ad409aa8063cbfec61cd1e"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "8491c80d59cc387e563dedf39a881128"
server_key_iv: "19dfec293056572d98bed851e71dd87b"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.861496 5689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:52:58.862082 5703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:58.862221 5689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:52:58.862298 5689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "6f7ae15be8ad409aa8063cbfec61cd1e"
format_stamp: "Formatted at 2026-09-27 02:52:58 on dist-test-slave-mfx9"
server_key: "8491c80d59cc387e563dedf39a881128"
server_key_iv: "19dfec293056572d98bed851e71dd87b"
server_key_version: "encryptionkey@0"
I20260927 02:52:58.862355 5689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:58.877547 5689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:58.877805 5689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:58.877928 5689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:58.881690 5689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42773
I20260927 02:52:58.881716 5755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42773 every 8 connection(s)
I20260927 02:52:58.882030 5689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:58.882604 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:58.883646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 5689
I20260927 02:52:58.883752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:58.883909 392 external_mini_cluster.cc:1428] Setting key aebbe22773e612547c17c7d9b0a23b02
I20260927 02:52:58.885768 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e: Bootstrap starting.
I20260927 02:52:58.886298 5756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:58.886559 5756 log.cc:824] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:58.887280 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e: No bootstrap required, opened a new log
I20260927 02:52:58.888974 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.884433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.0.98.62:42773)
0927 02:52:58.884601 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.884608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.884631 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:52:58.885068 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.885075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.885136 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.885246 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:58.885440 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.886313 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.887260 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.887677 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.887777 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.888384 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:58.888388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:58.888388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:58.888531 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:58.888561 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:58.888571 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:58.888624 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:58.888703 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:52:58.888812 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260927 02:52:58.889213 5756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ae15be8ad409aa8063cbfec61cd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } }
I20260927 02:52:58.889348 5756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:58.889369 5756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7ae15be8ad409aa8063cbfec61cd1e, State: Initialized, Role: FOLLOWER
I20260927 02:52:58.889479 5756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ae15be8ad409aa8063cbfec61cd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } }
I20260927 02:52:58.889557 5756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:58.889596 5756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:58.889639 5756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:58.889885 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:52:58.890192 5756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ae15be8ad409aa8063cbfec61cd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } }
I20260927 02:52:58.890278 5756 leader_election.cc:302] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7ae15be8ad409aa8063cbfec61cd1e; no voters:
I20260927 02:52:58.890411 5756 leader_election.cc:288] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:58.890457 5761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:58.890650 5756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:58.890807 5761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [term 1 LEADER]: Becoming Leader. State: Replica: 6f7ae15be8ad409aa8063cbfec61cd1e, State: Running, Role: LEADER
I20260927 02:52:58.890894 5761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ae15be8ad409aa8063cbfec61cd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } }
I20260927 02:52:58.891218 5763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7ae15be8ad409aa8063cbfec61cd1e. Latest consensus state: current_term: 1 leader_uuid: "6f7ae15be8ad409aa8063cbfec61cd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ae15be8ad409aa8063cbfec61cd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } } }
I20260927 02:52:58.891214 5762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f7ae15be8ad409aa8063cbfec61cd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ae15be8ad409aa8063cbfec61cd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42773 } } }
I20260927 02:52:58.891316 5763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:58.891320 5762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:58.891508 5770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:58.891955 5770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:58.893287 5770 catalog_manager.cc:1347] Generated new cluster ID: 8b7a0b3826ab47f7ba78810f79573338
I20260927 02:52:58.893333 5770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:58.898059 5770 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:58.898527 5770 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:58.904556 5770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f7ae15be8ad409aa8063cbfec61cd1e: Generated new TSK 0
I20260927 02:52:58.904716 5770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:52:58.920892 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:58.917874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.0.98.62:42773)
0927 02:52:58.917923 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:58.917925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:58.918012 (+ 87us) server_negotiation.cc:407] Connection header received
0927 02:52:58.918074 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:58.918075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:58.918086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:58.918141 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:52:58.918259 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.918735 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.919503 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.919688 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.919740 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.919935 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:52:58.920707 (+ 772us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:52:58.920825 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 02:52:58.920949 5759 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 02:52:58.920974 5785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:52:58.917836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42773 (local address 127.0.0.1:48436)
0927 02:52:58.917952 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:58.917961 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:58.918020 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:58.918146 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:58.918146 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:58.918148 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:52:58.918198 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:58.918200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.918829 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.918830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:58.919432 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:58.919434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:58.919797 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:58.919800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:58.919846 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:58.919851 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:52:58.920796 (+ 945us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:52:58.920879 (+ 83us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42773: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":82,"threads_started":1}
I20260927 02:52:58.921325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 5689
E20260927 02:52:58.932025 5786 webserver.cc:533] Webserver: sslize failed:
E20260927 02:52:58.932080 5786 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T02:52:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (195 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T02:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:52:58Z Disabled control of system clock
I20260927 02:52:58.964601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43655
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43655
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/testchainca.cert with env {}
W20260927 02:52:59.040697 5790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:52:59.040889 5790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:52:59.040918 5790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:52:59.042068 5790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:52:59.042105 5790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:52:59.042121 5790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:52:59.042136 5790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:52:59.042150 5790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:52:59.042166 5790 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:52:59.042184 5790 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 02:52:59.043638 5790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43655
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:52:59.043870 5790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:52:59.044057 5790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:52:59.048518 5798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:59.048478 5796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:52:59.048497 5795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:52:59.048588 5790 server_base.cc:1034] running on GCE node
I20260927 02:52:59.048897 5790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:52:59.049154 5790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:52:59.050271 5790 hybrid_clock.cc:648] HybridClock initialized: now 1790477579050257 us; error 31 us; skew 500 ppm
I20260927 02:52:59.051414 5790 webserver.cc:460] Webserver started at http://127.0.98.62:42911/ using document root <none> and password file <none>
I20260927 02:52:59.051609 5790 fs_manager.cc:359] Metadata directory not provided
I20260927 02:52:59.051652 5790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:52:59.051749 5790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:52:59.052726 5790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "3a02de6caf8e4736b501a605b8df6be8"
format_stamp: "Formatted at 2026-09-27 02:52:59 on dist-test-slave-mfx9"
server_key: "72e3f9fceb15de666bf0ca95c8dfb85e"
server_key_iv: "1efc8be88ba39ae371547eb90a28ca9d"
server_key_version: "encryptionkey@0"
I20260927 02:52:59.053030 5790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a02de6caf8e4736b501a605b8df6be8"
format_stamp: "Formatted at 2026-09-27 02:52:59 on dist-test-slave-mfx9"
server_key: "72e3f9fceb15de666bf0ca95c8dfb85e"
server_key_iv: "1efc8be88ba39ae371547eb90a28ca9d"
server_key_version: "encryptionkey@0"
I20260927 02:52:59.054126 5790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:52:59.054736 5804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:52:59.054893 5790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:52:59.054976 5790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "3a02de6caf8e4736b501a605b8df6be8"
format_stamp: "Formatted at 2026-09-27 02:52:59 on dist-test-slave-mfx9"
server_key: "72e3f9fceb15de666bf0ca95c8dfb85e"
server_key_iv: "1efc8be88ba39ae371547eb90a28ca9d"
server_key_version: "encryptionkey@0"
I20260927 02:52:59.055034 5790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:52:59.063081 5790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:52:59.063319 5790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:52:59.063433 5790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:52:59.066952 5790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43655
I20260927 02:52:59.067003 5856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43655 every 8 connection(s)
I20260927 02:52:59.067289 5790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:52:59.068238 5857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:52:59.068318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 5790
I20260927 02:52:59.068420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:52:59.068626 392 external_mini_cluster.cc:1428] Setting key 58c9d3d6c13ff44c41dae0bfe2f59274
I20260927 02:52:59.071394 5857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8: Bootstrap starting.
I20260927 02:52:59.072081 5857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8: Neither blocks nor log segments found. Creating new log.
I20260927 02:52:59.072382 5857 log.cc:824] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:52:59.073383 5857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8: No bootstrap required, opened a new log
I20260927 02:52:59.073838 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:59.069078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.0.98.62:43655)
0927 02:52:59.069239 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:59.069245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:59.069268 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:52:59.069683 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:59.069689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:59.069746 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:59.069850 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:59.070079 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:59.070981 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:59.072008 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:59.072482 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:59.072583 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:59.073232 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:52:59.073235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:52:59.073235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:52:59.073379 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:52:59.073408 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:52:59.073418 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:52:59.073478 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:59.073575 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 02:52:59.073633 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260927 02:52:59.074725 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:52:59.075492 5857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a02de6caf8e4736b501a605b8df6be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } }
I20260927 02:52:59.075652 5857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:52:59.075696 5857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a02de6caf8e4736b501a605b8df6be8, State: Initialized, Role: FOLLOWER
I20260927 02:52:59.075795 5857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a02de6caf8e4736b501a605b8df6be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } }
I20260927 02:52:59.075852 5857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:52:59.075874 5857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:52:59.075915 5857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:52:59.076395 5857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a02de6caf8e4736b501a605b8df6be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } }
I20260927 02:52:59.076481 5857 leader_election.cc:302] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a02de6caf8e4736b501a605b8df6be8; no voters:
I20260927 02:52:59.076602 5857 leader_election.cc:288] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:52:59.076640 5862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:52:59.076771 5862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [term 1 LEADER]: Becoming Leader. State: Replica: 3a02de6caf8e4736b501a605b8df6be8, State: Running, Role: LEADER
I20260927 02:52:59.076833 5857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:52:59.076849 5862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a02de6caf8e4736b501a605b8df6be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } }
I20260927 02:52:59.077198 5864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a02de6caf8e4736b501a605b8df6be8. Latest consensus state: current_term: 1 leader_uuid: "3a02de6caf8e4736b501a605b8df6be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a02de6caf8e4736b501a605b8df6be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } }
I20260927 02:52:59.077203 5863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a02de6caf8e4736b501a605b8df6be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a02de6caf8e4736b501a605b8df6be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } }
I20260927 02:52:59.077322 5864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:59.077328 5863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:52:59.077541 5869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:52:59.078009 5869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:52:59.079339 5869 catalog_manager.cc:1347] Generated new cluster ID: 00ae0f9430d1402caeefd9a889d81ba9
I20260927 02:52:59.079388 5869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:52:59.102100 5869 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:52:59.102592 5869 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:52:59.108139 5869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a02de6caf8e4736b501a605b8df6be8: Generated new TSK 0
I20260927 02:52:59.108296 5869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:52:59.149833 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:59.115540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.0.98.62:43655)
0927 02:52:59.115582 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:59.115584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:59.115679 (+ 95us) server_negotiation.cc:407] Connection header received
0927 02:52:59.115746 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:59.115747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:59.115757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:59.115811 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:52:59.115921 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:59.116406 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:59.117488 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:59.117665 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:59.117703 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:59.117876 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:52:59.149538 (+ 31662us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:52:59.149752 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51034: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:52:59.149873 5886 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:52:59.115467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43655 (local address 127.0.0.1:51034)
0927 02:52:59.115610 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:52:59.115620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:52:59.115685 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:52:59.115813 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:52:59.115814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:52:59.115816 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:52:59.115871 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:59.115874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:59.116473 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:59.116474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:59.117412 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:52:59.117415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:52:59.117723 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:52:59.117727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:52:59.117792 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:59.117797 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:52:59.149686 (+ 31889us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:52:59.149794 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43655: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260927 02:52:59.149914 5860 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51034: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260927 02:52:59.150192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 5790
2026-09-27T02:52:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (208 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (82 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:52:59 dist-test-slave-mfx9 krb5kdc[5898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:52:59 dist-test-slave-mfx9 krb5kdc[5898](info): set up 2 sockets
Sep 27 02:52:59 dist-test-slave-mfx9 krb5kdc[5898](info): commencing operation
krb5kdc: starting...
W20260927 02:53:00.254714 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:53:00.365626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41375
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:00.442559 5911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:00.442775 5911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:00.442812 5911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:00.443876 5911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:00.443907 5911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:00.443919 5911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:00.443930 5911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:00.443954 5911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:00.445343 5911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:00.445521 5911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:00.445684 5911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:00.450083 5919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:00.450168 5911 server_base.cc:1034] running on GCE node
W20260927 02:53:00.450129 5917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:00.450141 5916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:00.450465 5911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:00.450711 5911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:00.451862 5911 hybrid_clock.cc:648] HybridClock initialized: now 1790477580451851 us; error 26 us; skew 500 ppm
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:00.454000 5911 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:53:00.454397 5911 webserver.cc:460] Webserver started at http://127.0.98.62:46619/ using document root <none> and password file <none>
I20260927 02:53:00.454589 5911 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:00.454636 5911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:00.454773 5911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:00.455804 5911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "4bd6cd63f6d84f3aa2fe13224a239e28"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "f6a559cf3cf0999db50a5ec27d7dc44e"
server_key_iv: "6864d9bdf00c7f459492426427e7a18a"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.456135 5911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "4bd6cd63f6d84f3aa2fe13224a239e28"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "f6a559cf3cf0999db50a5ec27d7dc44e"
server_key_iv: "6864d9bdf00c7f459492426427e7a18a"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.457369 5911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:53:00.458074 5926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.458248 5911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:00.458328 5911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "4bd6cd63f6d84f3aa2fe13224a239e28"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "f6a559cf3cf0999db50a5ec27d7dc44e"
server_key_iv: "6864d9bdf00c7f459492426427e7a18a"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.458387 5911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:00.469106 5911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:00.470723 5911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:00.470867 5911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:00.474524 5911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41375
I20260927 02:53:00.474598 5978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41375 every 8 connection(s)
I20260927 02:53:00.475039 5911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:53:00.475667 5979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:00.477876 5979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28: Bootstrap starting.
I20260927 02:53:00.478385 5979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:00.478621 5979 log.cc:824] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:00.479355 5979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28: No bootstrap required, opened a new log
I20260927 02:53:00.480036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 5911
I20260927 02:53:00.480137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:53:00.480315 392 external_mini_cluster.cc:1428] Setting key dc8f73e516dab3b79f2074e85757ee64
I20260927 02:53:00.481333 5979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd6cd63f6d84f3aa2fe13224a239e28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } }
I20260927 02:53:00.481482 5979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:00.481516 5979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd6cd63f6d84f3aa2fe13224a239e28, State: Initialized, Role: FOLLOWER
I20260927 02:53:00.481642 5979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd6cd63f6d84f3aa2fe13224a239e28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } }
I20260927 02:53:00.481732 5979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:53:00.481763 5979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:53:00.481809 5979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:00.482415 5979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd6cd63f6d84f3aa2fe13224a239e28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } }
I20260927 02:53:00.482504 5979 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd6cd63f6d84f3aa2fe13224a239e28; no voters:
I20260927 02:53:00.482642 5979 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:53:00.482688 5984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:00.482831 5984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd6cd63f6d84f3aa2fe13224a239e28, State: Running, Role: LEADER
I20260927 02:53:00.482906 5979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:00.482919 5984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd6cd63f6d84f3aa2fe13224a239e28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } }
I20260927 02:53:00.483258 5985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bd6cd63f6d84f3aa2fe13224a239e28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd6cd63f6d84f3aa2fe13224a239e28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } } }
I20260927 02:53:00.483332 5985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:00.483496 5986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd6cd63f6d84f3aa2fe13224a239e28. Latest consensus state: current_term: 1 leader_uuid: "4bd6cd63f6d84f3aa2fe13224a239e28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd6cd63f6d84f3aa2fe13224a239e28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } } }
I20260927 02:53:00.483566 5986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:00.483673 5991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:00.484145 5991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:53:00.485874 5991 catalog_manager.cc:1347] Generated new cluster ID: f8eeff21cafd4671a2792a193d83fb70
I20260927 02:53:00.485925 5991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:00.490475 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:00.480804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.98.62:41375)
0927 02:53:00.480950 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:00.480957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:00.480982 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:53:00.481299 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:00.481305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:00.481348 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:00.481458 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:00.482167 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.483389 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.484358 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.484833 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.484941 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:00.487709 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:00.487713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:00.487714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:00.487725 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:00.489160 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:00.489458 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:00.489461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:00.489461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:00.489556 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:00.489720 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:00.489720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:00.489721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:00.489958 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:00.490094 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:00.490183 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:53:00.490314 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260927 02:53:00.515911 5991 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:53:00.516673 5991 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:00.529572 5991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bd6cd63f6d84f3aa2fe13224a239e28: Generated new TSK 0
I20260927 02:53:00.529785 5991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:53:00.561697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:00.641059 6005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:00.641366 6005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:00.641415 6005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:00.642915 6005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:00.642956 6005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:00.642982 6005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:00.644367 6005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:00.644537 6005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:00.644704 6005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:00.649325 6011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:00.649490 6005 server_base.cc:1034] running on GCE node
W20260927 02:53:00.649325 6013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:00.649343 6010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:00.649772 6005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:00.650007 6005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:00.651139 6005 hybrid_clock.cc:648] HybridClock initialized: now 1790477580651128 us; error 29 us; skew 500 ppm
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:00.653501 6005 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:53:00.653961 6005 webserver.cc:460] Webserver started at http://127.0.98.1:46807/ using document root <none> and password file <none>
I20260927 02:53:00.654155 6005 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:00.654202 6005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:00.654325 6005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:00.655404 6005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9efab47af4444b3b5daff73fb1f5811"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "5c1d93761153d384be4ece42d575396c"
server_key_iv: "0d732d33486a38aba254f8ab74db59d5"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.655731 6005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9efab47af4444b3b5daff73fb1f5811"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "5c1d93761153d384be4ece42d575396c"
server_key_iv: "0d732d33486a38aba254f8ab74db59d5"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.656980 6005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:53:00.657699 6020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.657893 6005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:00.657967 6005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "d9efab47af4444b3b5daff73fb1f5811"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "5c1d93761153d384be4ece42d575396c"
server_key_iv: "0d732d33486a38aba254f8ab74db59d5"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.658033 6005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:00.678848 6005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:00.681007 6005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:00.681124 6005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:00.681347 6005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:00.681632 6005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:00.681664 6005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.681686 6005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:00.681708 6005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.687387 6005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42497
I20260927 02:53:00.687440 6133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42497 every 8 connection(s)
I20260927 02:53:00.687745 6005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:00.696434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 6005
I20260927 02:53:00.696483 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:00.688426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52995 (local address 127.0.98.62:41375)
0927 02:53:00.688485 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:00.688487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:00.688713 (+ 226us) server_negotiation.cc:407] Connection header received
0927 02:53:00.689594 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:00.689595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:00.689610 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:00.689675 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:00.690297 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.690814 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.691768 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.692001 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.692052 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:00.694015 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:00.694017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:00.694017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:00.694025 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:00.695306 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:00.695656 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:00.695658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:00.695658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:00.695715 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:00.695902 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:00.695903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:00.695903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:00.696009 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:00.696057 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:00.696305 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:53:00.696391 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:53:00.696527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:53:00.696615 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:00.688394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41375 (local address 127.0.98.1:52995)
0927 02:53:00.688568 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:00.688610 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:00.689498 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:00.689741 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:00.689744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:00.690016 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:00.690215 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:00.690219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.690904 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.690905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:00.691700 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:00.691702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.692062 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.692063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:00.692157 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:00.692613 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:00.692621 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:00.693905 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:00.695369 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:00.695374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:00.695375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:00.695579 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:00.695771 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:00.695771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:00.695772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:00.695859 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:00.696052 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:00.696055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:00.696166 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 02:53:00.696240 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:53:00.696386 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260927 02:53:00.696702 392 external_mini_cluster.cc:1428] Setting key 7637b95c3b79f9ae9464e468ff5f1346
I20260927 02:53:00.697147 6134 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41375
I20260927 02:53:00.697238 6134 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:00.697418 6134 heartbeater.cc:511] Master 127.0.98.62:41375 requested a full tablet report, sending...
I20260927 02:53:00.697914 5943 ts_manager.cc:194] Registered new tserver with Master: d9efab47af4444b3b5daff73fb1f5811 (127.0.98.1:42497)
I20260927 02:53:00.698483 5943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52995
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:53:00.721710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:00.799477 6139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:00.799680 6139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:00.799702 6139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:00.800849 6139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:00.800889 6139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:00.800913 6139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:00.802407 6139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:00.802634 6139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:00.802862 6139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:00.807741 6144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:00.807744 6145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:00.807886 6139 server_base.cc:1034] running on GCE node
W20260927 02:53:00.807742 6147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:00.808154 6139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:00.808350 6139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:00.809479 6139 hybrid_clock.cc:648] HybridClock initialized: now 1790477580809468 us; error 28 us; skew 500 ppm
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:00.815426 6139 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:53:00.815865 6139 webserver.cc:460] Webserver started at http://127.0.98.2:40783/ using document root <none> and password file <none>
I20260927 02:53:00.816071 6139 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:00.816109 6139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:00.816198 6139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:00.817188 6139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "f9c6b555dfd847759517fd08ff03bea4"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "e046987533047b92b3e718cb390ad900"
server_key_iv: "8469ab3e0c5b02ce34d0633632593929"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.817476 6139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f9c6b555dfd847759517fd08ff03bea4"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "e046987533047b92b3e718cb390ad900"
server_key_iv: "8469ab3e0c5b02ce34d0633632593929"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.818606 6139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:53:00.819319 6154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.819484 6139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.819574 6139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "f9c6b555dfd847759517fd08ff03bea4"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "e046987533047b92b3e718cb390ad900"
server_key_iv: "8469ab3e0c5b02ce34d0633632593929"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.819628 6139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:00.830672 6139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:00.832692 6139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:00.832806 6139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:00.832986 6139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:00.833246 6139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:00.833278 6139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.833309 6139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:00.833331 6139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.838466 6139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41635
I20260927 02:53:00.838532 6267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41635 every 8 connection(s)
I20260927 02:53:00.838816 6139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:00.845911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 6139
I20260927 02:53:00.846007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:53:00.846184 392 external_mini_cluster.cc:1428] Setting key ca6cb25f192e51b899cd32e11320f32a
I20260927 02:53:00.847710 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:00.839496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52477 (local address 127.0.98.62:41375)
0927 02:53:00.839553 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:00.839555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:00.839736 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:53:00.840686 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:00.840687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:00.840699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:00.840748 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:00.841333 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.841837 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.842807 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.842998 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.843054 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:00.844942 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:00.844943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:00.844943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:00.844950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:00.846295 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:00.846862 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:00.846863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:00.846863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:00.846924 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:00.847155 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:00.847156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:00.847156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:00.847277 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:00.847341 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:00.847554 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 02:53:00.847628 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:53:00.847800 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:00.839476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41375 (local address 127.0.98.2:52477)
0927 02:53:00.839631 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:00.839664 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:00.840587 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:00.840818 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:00.840822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:00.841060 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:00.841243 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:00.841247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.841913 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.841914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:00.842703 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:00.842705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.843064 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.843064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:00.843179 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:00.843609 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:00.843616 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:00.844852 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:00.846371 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:00.846377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:00.846379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:00.846682 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:00.846994 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:00.846994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:00.846994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:00.847100 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:00.847333 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:00.847336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:00.847451 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 02:53:00.847526 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:53:00.847637 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260927 02:53:00.848215 6268 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41375
I20260927 02:53:00.848308 6268 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:00.848471 6268 heartbeater.cc:511] Master 127.0.98.62:41375 requested a full tablet report, sending...
I20260927 02:53:00.848807 5943 ts_manager.cc:194] Registered new tserver with Master: f9c6b555dfd847759517fd08ff03bea4 (127.0.98.2:41635)
I20260927 02:53:00.849236 5943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52477
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:53:00.868680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:00.945849 6273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:00.946043 6273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:00.946072 6273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:00.947368 6273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:00.947433 6273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:00.947472 6273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:00.948961 6273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:00.949200 6273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:00.949401 6273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:00.954010 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:00.954011 6279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:00.954128 6273 server_base.cc:1034] running on GCE node
W20260927 02:53:00.954224 6278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:00.954447 6273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:00.954658 6273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:00.955812 6273 hybrid_clock.cc:648] HybridClock initialized: now 1790477580955795 us; error 30 us; skew 500 ppm
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:00.958216 6273 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:53:00.958631 6273 webserver.cc:460] Webserver started at http://127.0.98.3:43129/ using document root <none> and password file <none>
I20260927 02:53:00.958879 6273 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:00.958925 6273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:00.959043 6273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:00.960067 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "411251e14ff64763810fe4c6356b557a"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "9476978f43a79f9a66a0148cfa801585"
server_key_iv: "86e871c39c35ce07fec26750b6f11272"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.960384 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "411251e14ff64763810fe4c6356b557a"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "9476978f43a79f9a66a0148cfa801585"
server_key_iv: "86e871c39c35ce07fec26750b6f11272"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.961694 6273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:00.962337 6288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.962471 6273 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:00.962555 6273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "411251e14ff64763810fe4c6356b557a"
format_stamp: "Formatted at 2026-09-27 02:53:00 on dist-test-slave-mfx9"
server_key: "9476978f43a79f9a66a0148cfa801585"
server_key_iv: "86e871c39c35ce07fec26750b6f11272"
server_key_version: "encryptionkey@0"
I20260927 02:53:00.962618 6273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:00.977912 6273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:00.980068 6273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:00.980201 6273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:00.980404 6273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:00.980702 6273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:00.980734 6273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.980772 6273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:00.980796 6273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:00.986339 6273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42569
I20260927 02:53:00.986392 6401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42569 every 8 connection(s)
I20260927 02:53:00.986680 6273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:53:00 dist-test-slave-mfx9 krb5kdc[5898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:00.992725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 6273
I20260927 02:53:00.992799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:53:00.992951 392 external_mini_cluster.cc:1428] Setting key be5cbda5698db5b04c8a3ea6d0aa3faf
I20260927 02:53:00.995450 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:00.987413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51705 (local address 127.0.98.62:41375)
0927 02:53:00.987468 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:00.987470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:00.987656 (+ 186us) server_negotiation.cc:407] Connection header received
0927 02:53:00.988568 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:00.988569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:00.988581 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:00.988634 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:00.989229 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.989758 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.990716 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.990962 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.991013 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:00.992950 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:00.992952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:00.992953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:00.992960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:00.994304 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:00.994646 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:00.994647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:00.994647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:00.994695 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:00.994913 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:00.994914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:00.994914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:00.995016 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:00.995062 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:00.995314 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 02:53:00.995384 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:53:00.995553 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:00.987411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41375 (local address 127.0.98.3:51705)
0927 02:53:00.987546 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:00.987580 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:00.988469 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:00.988702 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:00.988706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:00.988958 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:00.989145 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:00.989149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.989841 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.989842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:00.990638 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:00.990640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:00.991038 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:00.991039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:00.991154 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:00.991572 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:00.991580 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:00.992870 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:00.994373 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:00.994378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:00.994380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:00.994591 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:00.994734 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:00.994734 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:00.994735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:00.994869 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:00.995061 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:00.995063 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:00.995178 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 02:53:00.995259 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:53:00.995378 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260927 02:53:00.995924 6402 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41375
I20260927 02:53:00.996023 6402 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:00.996162 6402 heartbeater.cc:511] Master 127.0.98.62:41375 requested a full tablet report, sending...
I20260927 02:53:00.996474 5943 ts_manager.cc:194] Registered new tserver with Master: 411251e14ff64763810fe4c6356b557a (127.0.98.3:42569)
I20260927 02:53:00.996848 5943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51705
I20260927 02:53:01.004597 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/test-admin.keytab.
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:01.019281 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 02:53:01.020866 6409 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:01.020106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.0.98.1:42497)
0927 02:53:01.020289 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:01.020293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:01.020304 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:53:01.020331 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:01.020333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:01.020346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:01.020416 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:01.020729 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45888: BlockingRecv error: recv got EOF from 127.0.0.1:45888 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
W20260927 02:53:01.020905 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:01.021772 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:01.022626 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:53:01.029762 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:01.023220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.98.1:42497)
0927 02:53:01.023275 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:01.023277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:01.023285 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:01.023306 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:01.023307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:01.023317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:01.023363 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:01.023728 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:01.024447 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:01.025129 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:01.025450 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:01.025507 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:01.027449 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:01.027457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:01.027458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:01.027465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:01.028737 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:01.029036 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:01.029039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:01.029040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:01.029091 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:01.029246 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:01.029247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:01.029247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:01.029377 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:01.029502 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:01.029580 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:53:01.029658 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:53:01.031394 6409 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:01.030973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.0.98.1:42497)
0927 02:53:01.031026 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:01.031028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:01.031036 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:01.031079 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:01.031080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:01.031088 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:01.031128 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:01.031328 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45910: BlockingRecv error: recv got EOF from 127.0.0.1:45910 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 02:53:01.031510 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5cc)
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:01.032452 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:01.033270 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:53:01.039296 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:01.033704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.0.98.1:42497)
0927 02:53:01.033750 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:01.033751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:01.033796 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:53:01.033838 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:01.033839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:01.033845 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:01.033886 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:01.034165 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:01.034642 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:01.035304 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:01.035496 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:01.035544 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:01.037402 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:01.037402 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:01.037403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:01.037407 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:01.038458 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:01.038718 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:01.038719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:01.038719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:01.038812 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:01.038948 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:01.038949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:01.038949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:01.039029 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:01.039091 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:01.039159 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:53:01.039214 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 02:53:01.040583 6409 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:01.040183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.0.98.1:42497)
0927 02:53:01.040216 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:01.040218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:01.040239 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:53:01.040299 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:01.040299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:01.040306 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:01.040343 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:01.040541 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45924: BlockingRecv error: recv got EOF from 127.0.0.1:45924 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 02:53:01.040668 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477553417353-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:01.041479 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:01.042245 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[5898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:53:01.048511 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:01.042685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.0.98.1:42497)
0927 02:53:01.042735 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:01.042736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:01.042746 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:53:01.042828 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:01.042828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:01.042836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:01.042881 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:01.043186 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:01.043624 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:01.044240 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:01.044418 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:01.044469 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:01.046275 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:01.046276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:01.046276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:01.046282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:01.047711 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:01.047974 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:01.047975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:01.047975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:01.048021 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:01.048154 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:01.048154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:01.048154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:01.048230 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:01.048290 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:01.048376 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 02:53:01.048426 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:53:01.049018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 6005
I20260927 02:53:01.053377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 6139
I20260927 02:53:01.056723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 6273
I20260927 02:53:01.060184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 5911
2026-09-27T02:53:01Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1745 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[6422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[6422](info): set up 2 sockets
Sep 27 02:53:01 dist-test-slave-mfx9 krb5kdc[6422](info): commencing operation
krb5kdc: starting...
W20260927 02:53:02.002628 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:53:02.113214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34937
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:02.190083 6435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:02.190294 6435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:02.190323 6435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:02.191484 6435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:02.191522 6435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:02.191540 6435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:02.191556 6435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:02.191570 6435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:02.192991 6435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:02.193221 6435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:02.193419 6435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:02.197966 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:02.197985 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:02.198096 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.198136 6435 server_base.cc:1034] running on GCE node
I20260927 02:53:02.198463 6435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:02.198724 6435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:02.199915 6435 hybrid_clock.cc:648] HybridClock initialized: now 1790477582199902 us; error 28 us; skew 500 ppm
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:02.202275 6435 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:53:02.202666 6435 webserver.cc:460] Webserver started at http://127.0.98.62:36249/ using document root <none> and password file <none>
I20260927 02:53:02.202894 6435 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:02.202942 6435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:02.203058 6435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:02.204099 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "4c01d122e7eb49a3a9b0f298da539414"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "293c619b81fe48b4bdaac4a16aaa5f06"
server_key_iv: "d30c1a5bced12b627588fa3e8872dd7e"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.204409 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c01d122e7eb49a3a9b0f298da539414"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "293c619b81fe48b4bdaac4a16aaa5f06"
server_key_iv: "d30c1a5bced12b627588fa3e8872dd7e"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.205615 6435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:02.206295 6450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.206449 6435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:02.206532 6435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "4c01d122e7eb49a3a9b0f298da539414"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "293c619b81fe48b4bdaac4a16aaa5f06"
server_key_iv: "d30c1a5bced12b627588fa3e8872dd7e"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.206597 6435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:02.222046 6435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:02.224308 6435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:02.224444 6435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:02.228092 6435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34937
I20260927 02:53:02.228119 6502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34937 every 8 connection(s)
I20260927 02:53:02.228444 6435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:53:02.229333 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:02.231545 6503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: Bootstrap starting.
I20260927 02:53:02.232074 6503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:02.232318 6503 log.cc:824] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:02.232961 6503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: No bootstrap required, opened a new log
I20260927 02:53:02.234457 6503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } }
I20260927 02:53:02.234589 6503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:02.234618 6503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c01d122e7eb49a3a9b0f298da539414, State: Initialized, Role: FOLLOWER
I20260927 02:53:02.234742 6503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } }
I20260927 02:53:02.234831 6503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:53:02.234869 6503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:53:02.234915 6503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:02.235649 6503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } }
I20260927 02:53:02.235739 6503 leader_election.cc:302] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c01d122e7eb49a3a9b0f298da539414; no voters:
I20260927 02:53:02.235886 6503 leader_election.cc:288] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:53:02.235922 6506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:02.236063 6506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 1 LEADER]: Becoming Leader. State: Replica: 4c01d122e7eb49a3a9b0f298da539414, State: Running, Role: LEADER
I20260927 02:53:02.236111 6503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:02.236158 6506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } }
I20260927 02:53:02.236531 6508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c01d122e7eb49a3a9b0f298da539414. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } }
I20260927 02:53:02.236613 6508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:02.236850 6515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:02.236958 6507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } }
I20260927 02:53:02.237038 6507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:02.237389 6515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:53:02.237435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 6435
I20260927 02:53:02.237524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:53:02.237690 392 external_mini_cluster.cc:1428] Setting key 03164bb1abd4629e9780ee8b4080752c
I20260927 02:53:02.238952 6515 catalog_manager.cc:1347] Generated new cluster ID: c5f4a13495f442cea6e805af7caf8b79
I20260927 02:53:02.239003 6515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:02.246451 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.238207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.0.98.62:34937)
0927 02:53:02.238392 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.238401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.238457 (+ 56us) server_negotiation.cc:407] Connection header received
0927 02:53:02.238583 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.238590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.238644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.238809 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.239417 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.240308 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.241066 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.241472 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.241564 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.243635 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.243639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.243639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.243651 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.245082 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.245372 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.245375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.245375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.245470 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.245642 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.245643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.245643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.245856 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.245994 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.246083 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.246288 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260927 02:53:02.249028 6515 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:53:02.249696 6515 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:02.255759 6515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: Generated new TSK 0
I20260927 02:53:02.255913 6515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:53:02.283142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34937
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:02.361243 6529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:02.361446 6529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:02.361469 6529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:02.362617 6529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:02.362655 6529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:02.362679 6529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:02.364099 6529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:02.364300 6529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:02.364507 6529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:02.369158 6534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.369262 6529 server_base.cc:1034] running on GCE node
W20260927 02:53:02.369150 6535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:02.369154 6537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.369551 6529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:02.369771 6529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:02.370913 6529 hybrid_clock.cc:648] HybridClock initialized: now 1790477582370896 us; error 27 us; skew 500 ppm
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:02.373240 6529 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:53:02.373626 6529 webserver.cc:460] Webserver started at http://127.0.98.1:34169/ using document root <none> and password file <none>
I20260927 02:53:02.373822 6529 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:02.373867 6529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:02.373982 6529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:02.375061 6529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "7a9748b30bac15fe5ba1f4f5990d8471"
server_key_iv: "cce28fc4efd66908a1070cf1a1e8fadf"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.375384 6529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "7a9748b30bac15fe5ba1f4f5990d8471"
server_key_iv: "cce28fc4efd66908a1070cf1a1e8fadf"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.376559 6529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:02.377254 6544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.377388 6529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:02.377466 6529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "7a9748b30bac15fe5ba1f4f5990d8471"
server_key_iv: "cce28fc4efd66908a1070cf1a1e8fadf"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.377522 6529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:02.395357 6529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:02.397024 6529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:02.397151 6529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:02.397357 6529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:02.397634 6529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:02.397665 6529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.397692 6529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:02.397709 6529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.403038 6529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41989
I20260927 02:53:02.403093 6657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41989 every 8 connection(s)
I20260927 02:53:02.403407 6529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:53:02.407482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 6529
I20260927 02:53:02.407569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:53:02.407760 392 external_mini_cluster.cc:1428] Setting key 50bd629921863fd4718bdedfb327ae5b
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:02.412598 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.403994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44529 (local address 127.0.98.62:34937)
0927 02:53:02.404073 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.404075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.404219 (+ 144us) server_negotiation.cc:407] Connection header received
0927 02:53:02.404700 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.404701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.404716 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.404763 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.405669 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.406193 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.407154 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.407385 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.407437 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.409844 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.409846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.409846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.409854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.411313 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.411657 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.411658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.411659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.411720 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.411996 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.411997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.411998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.412132 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.412194 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.412419 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.412506 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:53:02.412689 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.403969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.1:44529)
0927 02:53:02.404102 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.404137 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.404607 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.404837 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.404840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.405393 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.405578 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.405583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.406274 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.406275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.407077 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.407079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.407450 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.407451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.407577 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.408155 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.408166 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.409739 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.411380 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.411385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.411387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.411593 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.411784 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.411784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.411785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.411934 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.412191 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.412194 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.412308 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.412382 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.412505 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260927 02:53:02.413203 6658 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34937
I20260927 02:53:02.413295 6658 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:02.413470 6658 heartbeater.cc:511] Master 127.0.98.62:34937 requested a full tablet report, sending...
I20260927 02:53:02.414001 6467 ts_manager.cc:194] Registered new tserver with Master: 9ddafcfe8d294d8f9ba5d2f3df2d7208 (127.0.98.1:41989)
I20260927 02:53:02.414880 6467 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44529
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:53:02.430857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34937
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:02.507586 6663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:02.507776 6663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:02.507797 6663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:02.508880 6663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:02.508912 6663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:02.508936 6663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:02.510254 6663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:02.510418 6663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:02.510596 6663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:02.515111 6669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:02.515164 6671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:02.515117 6668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.515508 6663 server_base.cc:1034] running on GCE node
I20260927 02:53:02.515702 6663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:02.515890 6663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:02.517017 6663 hybrid_clock.cc:648] HybridClock initialized: now 1790477582517006 us; error 28 us; skew 500 ppm
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:02.519497 6663 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:53:02.519883 6663 webserver.cc:460] Webserver started at http://127.0.98.2:32957/ using document root <none> and password file <none>
I20260927 02:53:02.520069 6663 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:02.520114 6663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:02.520233 6663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:02.521229 6663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "473f94ce017041b89d4ad07958552cd1"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "fa53ede76c4b4de3f63fa583ac07b301"
server_key_iv: "0796ffc15ec10c818451e0262d574ed4"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.521554 6663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "473f94ce017041b89d4ad07958552cd1"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "fa53ede76c4b4de3f63fa583ac07b301"
server_key_iv: "0796ffc15ec10c818451e0262d574ed4"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.522711 6663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:53:02.523346 6678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.523488 6663 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.523564 6663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "473f94ce017041b89d4ad07958552cd1"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "fa53ede76c4b4de3f63fa583ac07b301"
server_key_iv: "0796ffc15ec10c818451e0262d574ed4"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.523623 6663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:02.550272 6663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:02.552338 6663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:02.552469 6663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:02.552680 6663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:02.552973 6663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:02.553007 6663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.553035 6663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:02.553058 6663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.558122 6663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43879
I20260927 02:53:02.558187 6791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43879 every 8 connection(s)
I20260927 02:53:02.558452 6663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:02.565035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 6663
I20260927 02:53:02.565136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:53:02.565327 392 external_mini_cluster.cc:1428] Setting key d079c7cd466167c9dc158fa9862d992b
I20260927 02:53:02.567054 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.559173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60949 (local address 127.0.98.62:34937)
0927 02:53:02.559225 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.559227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.559398 (+ 171us) server_negotiation.cc:407] Connection header received
0927 02:53:02.559902 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.559904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.559916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.559965 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.560864 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.561361 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.562342 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.562537 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.562591 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.564503 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.564504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.564505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.564512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.565770 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.566169 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.566170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.566170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.566224 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.566467 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.566468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.566468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.566558 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.566603 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.566921 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.566992 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:53:02.567317 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.559148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.2:60949)
0927 02:53:02.559282 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.559320 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.559794 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.560037 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.560041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.560581 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.560766 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.560771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.561441 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.561442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.562249 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.562251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.562601 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.562602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.562718 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.563157 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.563165 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.564419 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.565881 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.565888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.565891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.566100 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.566282 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.566283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.566283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.566405 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.566605 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.566609 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.566780 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.566896 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.567049 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260927 02:53:02.567759 6792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34937
I20260927 02:53:02.567842 6792 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:02.568005 6792 heartbeater.cc:511] Master 127.0.98.62:34937 requested a full tablet report, sending...
I20260927 02:53:02.568358 6467 ts_manager.cc:194] Registered new tserver with Master: 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879)
I20260927 02:53:02.568701 6467 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60949
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:53:02.589208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34937
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:02.666812 6797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:02.666985 6797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:02.667006 6797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:02.668102 6797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:02.668138 6797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:02.668161 6797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:02.669535 6797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:02.669741 6797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:02.669942 6797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:02.674367 6803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:02.674412 6802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.674432 6797 server_base.cc:1034] running on GCE node
W20260927 02:53:02.674408 6805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.674803 6797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:02.675026 6797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:02.676157 6797 hybrid_clock.cc:648] HybridClock initialized: now 1790477582676146 us; error 28 us; skew 500 ppm
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:02.678268 6797 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:53:02.678680 6797 webserver.cc:460] Webserver started at http://127.0.98.3:36833/ using document root <none> and password file <none>
I20260927 02:53:02.678902 6797 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:02.678947 6797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:02.679069 6797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:02.680094 6797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "4639e9d88baa4f69b7b025dbfe161727"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "1eaf208be1f613c170c774222f1c8ce6"
server_key_iv: "750014ef9ab1d21085da9691a76367d0"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.680450 6797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4639e9d88baa4f69b7b025dbfe161727"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "1eaf208be1f613c170c774222f1c8ce6"
server_key_iv: "750014ef9ab1d21085da9691a76367d0"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.681595 6797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:53:02.682168 6812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.682298 6797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.682370 6797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "4639e9d88baa4f69b7b025dbfe161727"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "1eaf208be1f613c170c774222f1c8ce6"
server_key_iv: "750014ef9ab1d21085da9691a76367d0"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.682435 6797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:02.689558 6797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:02.691099 6797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:02.691214 6797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:02.691402 6797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:02.691655 6797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:02.691685 6797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.691712 6797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:02.691735 6797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.696852 6797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39103
I20260927 02:53:02.696887 6925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39103 every 8 connection(s)
I20260927 02:53:02.697152 6797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:02.703290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 6797
I20260927 02:53:02.703387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:53:02.703572 392 external_mini_cluster.cc:1428] Setting key 34850aa1cbdc39eb5aed5e080536a6cc
I20260927 02:53:02.705883 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.697798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57071 (local address 127.0.98.62:34937)
0927 02:53:02.697857 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.697859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.698028 (+ 169us) server_negotiation.cc:407] Connection header received
0927 02:53:02.698505 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.698507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.698518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.698568 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.699495 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.699986 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.700942 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.701142 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.701197 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.703100 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.703101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.703102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.703108 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.704629 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.705082 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.705083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.705084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.705145 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.705376 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.705377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.705377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.705515 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.705569 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.705765 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.705822 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:53:02.706154 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.697763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.3:57071)
0927 02:53:02.697920 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.697955 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.698416 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.698636 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.698640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.699227 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.699408 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.699412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.700062 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.700063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.700860 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.700862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.701204 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.701205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.701335 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.701727 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.701735 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.703012 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.704717 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.704723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.704725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.705009 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.705204 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.705204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.705204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.705320 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.705561 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.705565 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.705675 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.705752 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.705862 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260927 02:53:02.706534 6926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34937
I20260927 02:53:02.706615 6926 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:02.706749 6926 heartbeater.cc:511] Master 127.0.98.62:34937 requested a full tablet report, sending...
I20260927 02:53:02.707116 6467 ts_manager.cc:194] Registered new tserver with Master: 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103)
I20260927 02:53:02.707456 6467 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57071
I20260927 02:53:02.716111 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 02:53:02.739283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44457
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34937,127.0.98.61:44457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:02.819774 6931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:02.819981 6931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:02.820003 6931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:02.821105 6931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:02.821146 6931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:02.821161 6931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:02.821173 6931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:02.821185 6931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:02.822618 6931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34937,127.0.98.61:44457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
W20260927 02:53:02.822773 6931 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34937,127.0.98.61:44457'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 02:53:02.822923 6931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:02.823134 6931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:02.827744 6936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:02.827721 6939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.827777 6931 server_base.cc:1034] running on GCE node
W20260927 02:53:02.827739 6937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.828147 6931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:02.828397 6931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:02.829536 6931 hybrid_clock.cc:648] HybridClock initialized: now 1790477582829520 us; error 31 us; skew 500 ppm
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:02.831794 6931 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:53:02.832197 6931 webserver.cc:460] Webserver started at http://127.0.98.61:45781/ using document root <none> and password file <none>
I20260927 02:53:02.832381 6931 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:02.832424 6931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:02.832525 6931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:02.833540 6931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data/instance:
uuid: "b14b53aab66b4d2ba2bb21be1f0805d2"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "b5e2b859a99d63173735680f4a9239c8"
server_key_iv: "e3d243c98a447fff16b5789e58751fac"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.833834 6931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal/instance:
uuid: "b14b53aab66b4d2ba2bb21be1f0805d2"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "b5e2b859a99d63173735680f4a9239c8"
server_key_iv: "e3d243c98a447fff16b5789e58751fac"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.835052 6931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:53:02.835755 6946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.835949 6931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:53:02.836019 6931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
uuid: "b14b53aab66b4d2ba2bb21be1f0805d2"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "b5e2b859a99d63173735680f4a9239c8"
server_key_iv: "e3d243c98a447fff16b5789e58751fac"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.836081 6931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:02.850478 6931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:02.852713 6931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:02.852852 6931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:02.856839 6931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44457
I20260927 02:53:02.856892 6998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44457 every 8 connection(s)
I20260927 02:53:02.857204 6931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:53:02.858141 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:02.859611 6999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:53:02.863572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 6931
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:02.868300 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.860307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.0.98.62:34937)
0927 02:53:02.860373 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.860375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.860564 (+ 189us) server_negotiation.cc:407] Connection header received
0927 02:53:02.861018 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.861019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.861029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.861067 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.861632 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.862105 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.863200 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.863379 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.863441 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.865622 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.865623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.865623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.865631 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.866897 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.867307 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.867308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.867309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.867382 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.867611 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.867612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.867612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.867817 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.867896 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.868116 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.868198 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:53:02.868326 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.860241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:36928)
0927 02:53:02.860437 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.860471 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.860925 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.861143 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.861146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.861385 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.861568 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.861571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.862191 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.862192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.863128 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.863131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.863467 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.863468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.863616 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.864088 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.864098 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.865544 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.866973 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.866979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.866980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.867221 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.867458 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.867458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.867459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.867551 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.867889 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.867891 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.868003 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.868074 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.868178 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:53:02.869235 6999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:53:02.872236 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.864086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.0.98.61:44457)
0927 02:53:02.864226 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.864231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.864253 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:53:02.864284 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.864286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.864312 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.864399 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.864905 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.865588 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.866475 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.866776 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.866848 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.869483 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.869489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.869490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.869496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.871119 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.871443 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.871446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.871446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.871507 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.871690 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.871691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.871691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.871848 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.871984 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.872062 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.872138 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:53:02.875804 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.869457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.0.1:56078)
0927 02:53:02.869694 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.869703 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.869742 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.869889 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.869890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.870040 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.870080 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.870081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.870924 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.870925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.871445 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.871447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.871842 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.871843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.871899 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.872287 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.872291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.873351 (+ 1060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.875132 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.875133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.875133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.875269 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.875412 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.875412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.875412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.875454 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.875620 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.875621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.875648 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.875690 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.875750 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260927 02:53:02.875952 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.869504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.0.98.61:44457)
0927 02:53:02.869638 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.869640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.869733 (+ 93us) server_negotiation.cc:407] Connection header received
0927 02:53:02.869793 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.869794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.869805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.869881 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.870142 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.870853 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.871499 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.871775 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.871845 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.873416 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.873417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.873417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.873423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.875073 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.875322 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.875323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.875323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.875369 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.875487 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.875488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.875488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.875579 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.875625 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.875707 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.875825 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260927 02:53:02.876969 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Bootstrap starting.
I20260927 02:53:02.877527 6999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:02.877732 6999 log.cc:824] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:02.878326 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: No bootstrap required, opened a new log
I20260927 02:53:02.879639 6999 raft_consensus.cc:348] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } }
I20260927 02:53:02.879762 6999 raft_consensus.cc:374] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:02.879796 6999 raft_consensus.cc:729] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14b53aab66b4d2ba2bb21be1f0805d2, State: Initialized, Role: FOLLOWER
I20260927 02:53:02.879901 6999 consensus_queue.cc:260] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } }
I20260927 02:53:02.880178 6999 sys_catalog.cc:564] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:02.880129 7006 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } } }
I20260927 02:53:02.880223 7006 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:53:02.881693 7017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:53:02.881879 7017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:53:02.883177 6931 master_runner.cc:428] Detected that this master b14b53aab66b4d2ba2bb21be1f0805d2 is joining an existing cluster
I20260927 02:53:02.883215 6931 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:44457
I20260927 02:53:02.891314 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.884087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.0.98.61:44457)
0927 02:53:02.884122 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.884124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.884343 (+ 219us) server_negotiation.cc:407] Connection header received
0927 02:53:02.884415 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.884416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.884422 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.884444 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.884755 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.885506 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.886582 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.886902 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.886976 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.887854 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.887855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.887855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.887861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.889457 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.889831 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.889832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.889833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.889896 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.890823 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.890824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.890824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.890948 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.890987 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.891160 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.891228 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":15}
I20260927 02:53:02.891660 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.884141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.0.98.62:34937)
0927 02:53:02.884181 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.884183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.884233 (+ 50us) server_negotiation.cc:407] Connection header received
0927 02:53:02.884331 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.884332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.884344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.884397 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.884804 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.885526 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.887559 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.887866 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.887906 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.888967 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.888969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.888969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.888978 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.890545 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.890917 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.890918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.890918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.890982 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.891168 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.891169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.891169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.891284 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.891345 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.891493 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.891566 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:53:02.891671 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.884046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:36934)
0927 02:53:02.884181 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.884191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.884244 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.884398 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.884399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.884620 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.884685 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.884687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.886573 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.886574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.887461 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.887463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.887976 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.887977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.888050 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.888463 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.888468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.888878 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.890619 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.890619 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.890620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.890835 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.891039 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.891039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.891040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.891113 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.891341 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.891341 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.891395 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.891461 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.891560 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260927 02:53:02.892251 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.884185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.0.1:56080)
0927 02:53:02.884287 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.884296 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.884360 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.884479 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.884480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.884646 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.884691 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.884692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.885591 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.885592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.886481 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.886483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.886989 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.886990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.887043 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.887465 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.887469 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.887774 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.889540 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.889541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.889541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.889755 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.890637 (+ 882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.890638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.890639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.890726 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.891026 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.891026 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.891062 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.891647 (+ 585us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.892149 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":68,"threads_started":1}
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:53:02.902161 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.893899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.0.98.61:44457)
0927 02:53:02.893933 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.893935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.894954 (+ 1019us) server_negotiation.cc:407] Connection header received
0927 02:53:02.895031 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.895032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.895041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.895092 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.895562 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.895997 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.897075 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.897251 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.897314 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.899413 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.899415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.899416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.899423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.901038 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.901408 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.901409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.901409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.901462 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.901636 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.901637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.901637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.901747 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.901807 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.901979 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.902066 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:53:02.902180 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.893853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.0.1:56086)
0927 02:53:02.894847 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.894893 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.894965 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.895098 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.895101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.895424 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.895491 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.895492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.896063 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.896064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.896990 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.896992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.897321 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.897322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.897407 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.897827 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.897831 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.899329 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.901121 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.901129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.901132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.901340 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.901520 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.901521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.901521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.901583 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.901807 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.901814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.901871 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.901949 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.902069 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":73,"threads_started":1}
I20260927 02:53:02.902722 6467 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:44457: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 44457
}
attrs {
promote: true
}
}
dest_uuid: "4c01d122e7eb49a3a9b0f298da539414"
cas_config_opid_index: -1
I20260927 02:53:02.903086 6467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } }
I20260927 02:53:02.903558 7027 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b14b53aab66b4d2ba2bb21be1f0805d2 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } }
I20260927 02:53:02.903813 6973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:02.904641 7027 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:44457
I20260927 02:53:02.904698 6973 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Refusing update from remote peer 4c01d122e7eb49a3a9b0f298da539414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 02:53:02.904728 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } }
I20260927 02:53:02.904814 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:02.904997 6931 master.cc:537] Master@127.0.98.61:44457 shutting down...
I20260927 02:53:02.905112 7026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:53:02.905853 6452 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 02:53:02.906265 6452 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 -> Peer b14b53aab66b4d2ba2bb21be1f0805d2 (127.0.98.61:44457): Couldn't send request to peer b14b53aab66b4d2ba2bb21be1f0805d2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:53:02.908198 6931 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:02.908283 6931 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:02.908329 6931 tablet_replica.cc:330] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: stopping tablet replica
I20260927 02:53:02.920725 6931 master.cc:552] Master@127.0.98.61:44457 shutdown complete.
I20260927 02:53:02.920766 6931 master_runner.cc:305] Clearing existing system tablet
I20260927 02:53:02.920876 6931 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:02.920914 6931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:02.921437 6931 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:53:02.921856 7031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.921994 6931 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:02.922043 6931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.922065 6931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
uuid: "b14b53aab66b4d2ba2bb21be1f0805d2"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "b5e2b859a99d63173735680f4a9239c8"
server_key_iv: "e3d243c98a447fff16b5789e58751fac"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.922634 6931 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:02.924088 6931 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:53:02.924144 6931 log.cc:1196] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 02:53:02.924331 6931 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Deleting consensus metadata
I20260927 02:53:02.924429 6931 master_runner.cc:315] Copying system tablet from 127.0.98.62:34937
I20260927 02:53:02.925032 6931 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34937
I20260927 02:53:02.930603 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.925351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.0.98.62:34937)
0927 02:53:02.925395 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.925397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.925509 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:53:02.925593 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.925594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.925603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.925650 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.925953 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.926449 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.927243 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.927443 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.927499 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.928373 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.928375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.928375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.928382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.929592 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.929927 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.929928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.929928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.929983 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.930137 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.930137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.930138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.930239 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.930317 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.930443 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.930511 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:53:02.930617 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.925302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:36946)
0927 02:53:02.925448 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:02.925458 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:02.925524 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:02.925653 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:02.925653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:02.925823 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:02.925882 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.925884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.926516 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.926517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.927168 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:02.927170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.927502 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.927503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:02.927564 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.928011 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:02.928015 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:02.928294 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:02.929671 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.929672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.929672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.929863 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.930034 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:02.930035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:02.930035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:02.930089 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:02.930310 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:02.930311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:02.930356 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:53:02.930417 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:53:02.930512 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260927 02:53:02.931077 6487 tablet_copy_service.cc:140] P 4c01d122e7eb49a3a9b0f298da539414: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b14b53aab66b4d2ba2bb21be1f0805d2 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36946)
I20260927 02:53:02.931152 6487 tablet_copy_service.cc:161] P 4c01d122e7eb49a3a9b0f298da539414: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b14b53aab66b4d2ba2bb21be1f0805d2 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36946: session id = b14b53aab66b4d2ba2bb21be1f0805d2-00000000000000000000000000000000
I20260927 02:53:02.931747 6487 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:53:02.932305 6931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:53:02.933112 6931 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: tablet copy: Starting download of 0 data blocks...
I20260927 02:53:02.933226 6931 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: tablet copy: Starting download of 1 WAL segments...
I20260927 02:53:02.933943 6931 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:53:02.934631 6487 tablet_copy_service.cc:342] P 4c01d122e7eb49a3a9b0f298da539414: Request end of tablet copy session b14b53aab66b4d2ba2bb21be1f0805d2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36946
I20260927 02:53:02.934680 6487 tablet_copy_service.cc:433] P 4c01d122e7eb49a3a9b0f298da539414: ending tablet copy session b14b53aab66b4d2ba2bb21be1f0805d2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b14b53aab66b4d2ba2bb21be1f0805d2
I20260927 02:53:02.936906 6931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:02.937999 7038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:02.938102 7041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.938308 6931 server_base.cc:1034] running on GCE node
W20260927 02:53:02.938864 7039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:02.939006 6931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:02.939137 6931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:02.940274 6931 hybrid_clock.cc:648] HybridClock initialized: now 1790477582940262 us; error 27 us; skew 500 ppm
Sep 27 02:53:02 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:02.941641 6931 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:53:02.941918 6931 webserver.cc:460] Webserver started at http://127.0.98.61:45135/ using document root <none> and password file <none>
I20260927 02:53:02.942027 6931 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:02.942059 6931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:02.942575 6931 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.943092 7048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.943230 6931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:02.943285 6931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
uuid: "b14b53aab66b4d2ba2bb21be1f0805d2"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "b5e2b859a99d63173735680f4a9239c8"
server_key_iv: "e3d243c98a447fff16b5789e58751fac"
server_key_version: "encryptionkey@0"
I20260927 02:53:02.943336 6931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:02.963955 6931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:02.965689 6931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:02.968760 6931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44457
I20260927 02:53:02.968789 7100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44457 every 8 connection(s)
I20260927 02:53:02.969084 6931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:53:02.969779 7101 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:53:02.970157 7101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Bootstrap starting.
I20260927 02:53:02.972250 7101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:53:02.972414 7101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: Bootstrap complete.
I20260927 02:53:02.972561 7101 raft_consensus.cc:348] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } }
I20260927 02:53:02.972625 7101 raft_consensus.cc:729] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b14b53aab66b4d2ba2bb21be1f0805d2, State: Initialized, Role: LEARNER
I20260927 02:53:02.972657 7101 consensus_queue.cc:260] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } }
I20260927 02:53:02.972826 7101 sys_catalog.cc:564] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:02.972818 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } }
I20260927 02:53:02.972908 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:53:02.974237 7116 catalog_manager.cc:1259] Loaded cluster ID: c5f4a13495f442cea6e805af7caf8b79
I20260927 02:53:02.974747 7116 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: loading cluster ID for follower catalog manager: success
I20260927 02:53:02.976414 7116 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: acquiring CA information for follower catalog manager: success
I20260927 02:53:02.976799 7116 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:53:02.980293 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:02.973793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.0.98.61:44457)
0927 02:53:02.973879 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:02.973881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:02.973927 (+ 46us) server_negotiation.cc:407] Connection header received
0927 02:53:02.973950 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:02.973951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:02.973963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:02.974068 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:02.974576 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.975525 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.976203 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:02.976425 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:02.976477 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:02.977786 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:02.977787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:02.977787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:02.977794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:02.979189 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.979561 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.979562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.979562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.979608 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:02.979786 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:02.979786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:02.979787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:02.979874 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:02.979967 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:02.980140 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 02:53:02.980213 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":50,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 02:53:03.006064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40861
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34937,127.0.98.61:44457,127.0.98.60:40861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:03.087460 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:03.087663 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:03.087684 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:03.088822 7119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:03.088862 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:03.088876 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:03.088886 7119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:03.088909 7119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:03.090381 7119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34937,127.0.98.61:44457,127.0.98.60:40861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:03.090621 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:03.090862 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:03.095425 7125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.095479 7127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.095425 7124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:03.095479 7119 server_base.cc:1034] running on GCE node
I20260927 02:53:03.095835 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:03.096083 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:03.097208 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790477583097195 us; error 28 us; skew 500 ppm
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:03.099501 7119 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:53:03.099896 7119 webserver.cc:460] Webserver started at http://127.0.98.60:37213/ using document root <none> and password file <none>
I20260927 02:53:03.100095 7119 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:03.100140 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:03.100255 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:03.101336 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data/instance:
uuid: "30a21d18e77b47dc8d555a6f603c3807"
format_stamp: "Formatted at 2026-09-27 02:53:03 on dist-test-slave-mfx9"
server_key: "b13fded3263810ed672c5efb06af198e"
server_key_iv: "879183c5c2b65b1807cb317b65e637fb"
server_key_version: "encryptionkey@0"
I20260927 02:53:03.101667 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal/instance:
uuid: "30a21d18e77b47dc8d555a6f603c3807"
format_stamp: "Formatted at 2026-09-27 02:53:03 on dist-test-slave-mfx9"
server_key: "b13fded3263810ed672c5efb06af198e"
server_key_iv: "879183c5c2b65b1807cb317b65e637fb"
server_key_version: "encryptionkey@0"
I20260927 02:53:03.102906 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:03.103570 7134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.103727 7119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:03.103801 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
uuid: "30a21d18e77b47dc8d555a6f603c3807"
format_stamp: "Formatted at 2026-09-27 02:53:03 on dist-test-slave-mfx9"
server_key: "b13fded3263810ed672c5efb06af198e"
server_key_iv: "879183c5c2b65b1807cb317b65e637fb"
server_key_version: "encryptionkey@0"
I20260927 02:53:03.103865 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:03.115437 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:03.117101 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:03.117229 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:03.121063 7119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40861
I20260927 02:53:03.121104 7186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40861 every 8 connection(s)
I20260927 02:53:03.121420 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:53:03.122301 7187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:03.123778 7187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:03.130996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 7119
I20260927 02:53:03.132330 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.124609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.0.98.62:34937)
0927 02:53:03.124672 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.124673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.124726 (+ 53us) server_negotiation.cc:407] Connection header received
0927 02:53:03.125175 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.125176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.125187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.125231 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.125802 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.126330 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.127379 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.127567 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.127621 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.129598 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.129599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.129599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.129606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.130975 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.131405 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.131407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.131407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.131479 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.131746 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.131747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.131748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.131859 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.131914 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.132159 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.132241 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:53:03.132488 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.124456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:36952)
0927 02:53:03.124613 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.124653 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.125083 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.125291 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.125295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.125523 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.125699 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.125703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.126406 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.126406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.127299 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.127301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.127624 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.127625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.127718 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.128130 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.128137 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.129427 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.131042 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.131049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.131051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.131313 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.131562 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.131563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.131563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.131683 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.131914 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.131920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.132043 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.132128 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.132271 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260927 02:53:03.132982 7187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:53:03.139561 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.133209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.0.1:56122)
0927 02:53:03.133249 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.133259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.133297 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.133425 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.133425 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.133558 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.133602 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.133603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.134517 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.134517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.135221 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.135223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.135642 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.135643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.135698 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.136050 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.136054 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.137236 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.138604 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.138605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.138606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.138858 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.139052 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.139053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.139053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.139098 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.139325 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.139326 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.139358 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.139404 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.139500 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:53:03.139601 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.133280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.0.98.61:44457)
0927 02:53:03.133325 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.133327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.133336 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:03.133362 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.133362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.133371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.133418 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.133671 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.134428 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.135305 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.135563 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.135639 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.137317 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.137318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.137318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.137324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.138542 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.138920 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.138921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.138922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.138989 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.139143 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.139143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.139144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.139270 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.139337 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.139431 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.139513 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:53:03.139935 7187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:53:03.140657 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.131552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.0.98.60:40861)
0927 02:53:03.131703 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.131707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.131730 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:53:03.131752 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.131754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.131769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.131843 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.132594 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.133501 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.134555 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.134999 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.135066 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.137905 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.137911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.137912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.137918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.139540 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.139871 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.139874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.139874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.139925 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.140124 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.140125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.140125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.140264 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.140412 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.140475 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.140568 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:53:03.146044 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.140145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.0.1:57162)
0927 02:53:03.140181 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.140189 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.140218 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.140405 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.140405 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.140563 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.140585 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.140586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.141342 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.141343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.141798 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.141799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.142123 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.142124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.142204 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.142583 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.142588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.143668 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.145381 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.145382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.145382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.145521 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.145657 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.145657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.145657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.145700 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.145845 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.145845 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.145873 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.145914 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.145974 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:53:03.146550 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.140198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.98.60:40861)
0927 02:53:03.140321 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.140322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.140327 (+ 5us) server_negotiation.cc:407] Connection header received
0927 02:53:03.140339 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.140339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.140345 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.140397 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.140651 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.141289 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.141858 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.142045 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.142085 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.143735 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.143736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.143737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.143742 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.145307 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.145574 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.145575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.145575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.145616 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.145732 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.145733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.145733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.145808 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.145846 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.145930 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.146015 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260927 02:53:03.147706 7187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Bootstrap starting.
I20260927 02:53:03.148285 7187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:03.148535 7187 log.cc:824] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:03.149138 7187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: No bootstrap required, opened a new log
I20260927 02:53:03.150416 7187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } }
I20260927 02:53:03.150530 7187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:03.150560 7187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a21d18e77b47dc8d555a6f603c3807, State: Initialized, Role: FOLLOWER
I20260927 02:53:03.150676 7187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } }
I20260927 02:53:03.150935 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } } }
I20260927 02:53:03.150985 7187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:03.151028 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:53:03.152789 7204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:53:03.152841 7204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:53:03.154840 7119 master_runner.cc:428] Detected that this master 30a21d18e77b47dc8d555a6f603c3807 is joining an existing cluster
I20260927 02:53:03.154875 7119 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40861
I20260927 02:53:03.162827 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.155833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.0.98.60:40861)
0927 02:53:03.155861 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.155862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.155980 (+ 118us) server_negotiation.cc:407] Connection header received
0927 02:53:03.156022 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.156023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.156033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.156085 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.156493 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.157223 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158305 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.158567 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158611 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.160030 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.160032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.160032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.160038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.161634 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.162011 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.162012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.162013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.162086 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.162281 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.162281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.162282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.162436 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.162497 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.162647 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.162720 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:53:03.163082 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.155687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.0.1:57174)
0927 02:53:03.155912 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.155921 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.155978 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.156117 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.156118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.156352 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.156416 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.156418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.157322 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.157323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.158190 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.158192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.158699 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.158746 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.159159 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.159163 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.159939 (+ 776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.161716 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.161717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.161718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.161941 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.162139 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.162140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.162141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.162219 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.162519 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.162519 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.162554 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.162897 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.162991 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":69,"threads_started":1}
I20260927 02:53:03.186836 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.155718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.0.98.62:34937)
0927 02:53:03.155759 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.155761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.156097 (+ 336us) server_negotiation.cc:407] Connection header received
0927 02:53:03.156178 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.156178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.156188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.156215 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.156601 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.157319 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158391 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.158688 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158721 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.184180 (+ 25459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.184182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.184182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.184192 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.185623 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.185930 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.185932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.185932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.185990 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.186147 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.186148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.186148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.186280 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.186348 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.186634 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.186717 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:53:03.186975 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.155849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.0.98.61:44457)
0927 02:53:03.155891 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.155892 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.156004 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:53:03.156086 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.156086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.156095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.156123 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.156585 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.157324 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158385 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.158673 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158708 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.184141 (+ 25433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.184143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.184144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.184153 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.185865 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.186234 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.186235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.186235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.186285 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.186468 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.186468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.186469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.186564 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.186619 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.186825 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.186896 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:53:03.186857 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.155799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:36966)
0927 02:53:03.156054 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.156061 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.156116 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.156270 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.156270 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.156480 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.156526 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.156527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.157406 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.157407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.158309 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.158310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.158807 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.158883 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.183408 (+ 24525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.183414 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.184019 (+ 605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.185696 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.185697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.185698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.185868 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.186041 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.186042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.186042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.186099 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.186478 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.186479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.186533 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.186593 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.186703 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":41,"thread_start_us":59,"threads_started":1}
I20260927 02:53:03.187273 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.155712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.0.1:56138)
0927 02:53:03.155958 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.155966 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.156023 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.156189 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.156190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.156445 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.156504 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.156505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.157406 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.157407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.158294 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.158297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.158808 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.158808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.158886 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.183471 (+ 24585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.183474 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.184019 (+ 545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.185965 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.185967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.185967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.186164 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.186340 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.186341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.186341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.186409 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.186647 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.186648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.186688 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.187082 (+ 394us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.187176 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":26,"thread_start_us":63,"threads_started":1}
I20260927 02:53:03.191720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 6529
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:53:03.195389 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.189058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.0.98.60:40861)
0927 02:53:03.189090 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.189092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.189103 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:53:03.189177 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.189177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.189184 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.189224 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.189494 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.189926 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.190746 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.190956 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.191001 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.192914 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.192915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.192915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.192922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.194337 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.194683 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.194684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.194684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.194735 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.194920 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.194921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.194922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.195045 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.195103 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.195236 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.195305 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:53:03.195411 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.189025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.0.1:57176)
0927 02:53:03.189073 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.189082 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.189115 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.189229 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.189230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.189392 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.189438 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.189440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.189987 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.189988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.190682 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.190685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.191013 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.191014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.191085 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.191436 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.191440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.192819 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.194413 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.194413 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.194414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.194612 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.194804 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.194804 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.194805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.194864 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.195101 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.195102 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.195144 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.195208 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.195313 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:53:03.195850 6467 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40861: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40861
}
attrs {
promote: true
}
}
dest_uuid: "4c01d122e7eb49a3a9b0f298da539414"
cas_config_opid_index: 5
I20260927 02:53:03.196201 6467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } }
I20260927 02:53:03.196635 7026 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 30a21d18e77b47dc8d555a6f603c3807 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.197052 7161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:03.197455 7026 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40861
I20260927 02:53:03.197526 7026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.197583 7026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:03.197788 7119 master.cc:537] Master@127.0.98.60:40861 shutting down...
I20260927 02:53:03.198015 7161 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Refusing update from remote peer 4c01d122e7eb49a3a9b0f298da539414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:53:03.199882 7027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:03.200773 7119 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:03.200861 7119 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:03.200910 7119 tablet_replica.cc:330] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: stopping tablet replica
W20260927 02:53:03.201315 6453 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 -> Peer 30a21d18e77b47dc8d555a6f603c3807 (127.0.98.60:40861): Couldn't send request to peer 30a21d18e77b47dc8d555a6f603c3807. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:53:03.201467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41989
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34937
--webserver_port=34169
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457,127.0.98.60:40861 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:53:03.204873 7119 master.cc:552] Master@127.0.98.60:40861 shutdown complete.
I20260927 02:53:03.204917 7119 master_runner.cc:305] Clearing existing system tablet
I20260927 02:53:03.205030 7119 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:03.205061 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:03.205346 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.196715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.0.1:56152)
0927 02:53:03.196889 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.196898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.196934 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.200167 (+ 3233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.200168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.200170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:03.200212 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.200213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.202649 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.202650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.204286 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.204289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.205093 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.205094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.205158 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.205185 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.205231 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.205269 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 02:53:03.205581 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.196792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.0.98.61:44457)
0927 02:53:03.197940 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.197943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.197952 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:03.197975 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.197977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.197998 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.198060 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:03.200289 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.201051 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.204386 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.204927 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.204990 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.205061 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.205477 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.205521 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1124}
I20260927 02:53:03.205802 7075 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 LEARNER]: Refusing update from remote peer 4c01d122e7eb49a3a9b0f298da539414: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:53:03.206019 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:03.206000 7027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 02:53:03.206537 7075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 30a21d18e77b47dc8d555a6f603c3807 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.207636 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c01d122e7eb49a3a9b0f298da539414. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.207718 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:53:03.207751 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: true } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.207794 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:53:03.208009 7216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.208027 7027 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: attempting to promote NON_VOTER b14b53aab66b4d2ba2bb21be1f0805d2 to VOTER
I20260927 02:53:03.208101 7119 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:03.208154 7119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:53:03.208175 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
uuid: "30a21d18e77b47dc8d555a6f603c3807"
format_stamp: "Formatted at 2026-09-27 02:53:03 on dist-test-slave-mfx9"
server_key: "b13fded3263810ed672c5efb06af198e"
server_key_iv: "879183c5c2b65b1807cb317b65e637fb"
server_key_version: "encryptionkey@0"
I20260927 02:53:03.208220 7027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } }
I20260927 02:53:03.208827 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.208665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.0.1:57188)
0927 02:53:03.208763 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.208801 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 02:53:03.208917 7075 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 LEARNER]: Refusing update from remote peer 4c01d122e7eb49a3a9b0f298da539414: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 02:53:03.208962 7119 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:03.209093 7026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 02:53:03.209955 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.209867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.0.1:57190)
0927 02:53:03.209908 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.209930 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:53:03.210095 6453 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 -> Peer 30a21d18e77b47dc8d555a6f603c3807 (127.0.98.60:40861): Couldn't send request to peer 30a21d18e77b47dc8d555a6f603c3807. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40861: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:53:03.210248 7119 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:53:03.210307 7119 log.cc:1196] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 02:53:03.210492 7119 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Deleting consensus metadata
I20260927 02:53:03.210444 7026 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b14b53aab66b4d2ba2bb21be1f0805d2 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.210812 7119 master_runner.cc:315] Copying system tablet from 127.0.98.62:34937
I20260927 02:53:03.210968 7075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b14b53aab66b4d2ba2bb21be1f0805d2 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.211367 7119 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34937
I20260927 02:53:03.211810 7217 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.211884 7217 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:53:03.212638 7026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.212719 7026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:03.212767 7026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.212812 7026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:03.221649 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.215229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36976 (local address 127.0.98.62:34937)
0927 02:53:03.215266 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.215268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.215400 (+ 132us) server_negotiation.cc:407] Connection header received
0927 02:53:03.215446 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.215447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.215458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.215506 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.215821 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.216598 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.217589 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.217888 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.217944 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.219026 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.219027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.219028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.219035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.220580 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.220931 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.220932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.220932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.220984 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.221164 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.221165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.221165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.221289 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.221362 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.221489 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.221561 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:53:03.221658 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.215180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:36976)
0927 02:53:03.215324 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.215334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.215394 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.215516 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.215517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.215702 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.215763 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.215765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.216675 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.216676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.217516 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.217519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.217961 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.217962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.218031 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.218527 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.218532 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.218934 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.220660 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.220661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.220661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.220857 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.221041 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.221041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.221042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.221104 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.221351 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.221352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.221395 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.221462 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.221569 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260927 02:53:03.222100 6487 tablet_copy_service.cc:140] P 4c01d122e7eb49a3a9b0f298da539414: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 30a21d18e77b47dc8d555a6f603c3807 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36976)
I20260927 02:53:03.222153 6487 tablet_copy_service.cc:161] P 4c01d122e7eb49a3a9b0f298da539414: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 30a21d18e77b47dc8d555a6f603c3807 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36976: session id = 30a21d18e77b47dc8d555a6f603c3807-00000000000000000000000000000000
I20260927 02:53:03.222683 6487 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:53:03.223204 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:53:03.224440 7119 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: tablet copy: Starting download of 0 data blocks...
I20260927 02:53:03.224589 7119 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: tablet copy: Starting download of 1 WAL segments...
I20260927 02:53:03.225240 7119 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:53:03.226073 6487 tablet_copy_service.cc:342] P 4c01d122e7eb49a3a9b0f298da539414: Request end of tablet copy session 30a21d18e77b47dc8d555a6f603c3807-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36976
I20260927 02:53:03.226126 6487 tablet_copy_service.cc:433] P 4c01d122e7eb49a3a9b0f298da539414: ending tablet copy session 30a21d18e77b47dc8d555a6f603c3807-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 30a21d18e77b47dc8d555a6f603c3807
I20260927 02:53:03.228327 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:03.229487 7229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.229751 7227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:03.229943 7119 server_base.cc:1034] running on GCE node
W20260927 02:53:03.231472 7226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:03.231642 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:03.231767 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:03.232884 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790477583232890 us; error 46 us; skew 500 ppm
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:03.234316 7119 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:53:03.234555 7119 webserver.cc:460] Webserver started at http://127.0.98.60:46451/ using document root <none> and password file <none>
I20260927 02:53:03.234656 7119 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:03.234686 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:03.235387 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.235931 7236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.236084 7119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.236133 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
uuid: "30a21d18e77b47dc8d555a6f603c3807"
format_stamp: "Formatted at 2026-09-27 02:53:03 on dist-test-slave-mfx9"
server_key: "b13fded3263810ed672c5efb06af198e"
server_key_iv: "879183c5c2b65b1807cb317b65e637fb"
server_key_version: "encryptionkey@0"
I20260927 02:53:03.236188 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:03.251586 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:03.253283 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:03.256131 7119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40861
I20260927 02:53:03.256490 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:53:03.257313 7289 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:53:03.257805 7289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Bootstrap starting.
I20260927 02:53:03.258865 7288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40861 every 8 connection(s)
I20260927 02:53:03.260681 7289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:53:03.260869 7289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Bootstrap complete.
I20260927 02:53:03.261059 7289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } }
I20260927 02:53:03.261144 7289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30a21d18e77b47dc8d555a6f603c3807, State: Initialized, Role: LEARNER
I20260927 02:53:03.261178 7289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } }
I20260927 02:53:03.261343 7289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:03.261348 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.261550 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:53:03.262682 7303 catalog_manager.cc:1259] Loaded cluster ID: c5f4a13495f442cea6e805af7caf8b79
I20260927 02:53:03.262782 7303 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: loading cluster ID for follower catalog manager: success
I20260927 02:53:03.263906 7303 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: acquiring CA information for follower catalog manager: success
I20260927 02:53:03.264196 7303 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 02:53:03.298480 7213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:03.298673 7213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:03.298704 7213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:03.299857 7213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:03.299898 7213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:03.299932 7213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:03.301352 7213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34169
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457,127.0.98.60:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:03.301570 7213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:03.301769 7213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:03.306391 7309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.306403 7308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.306411 7311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:03.306828 7213 server_base.cc:1034] running on GCE node
I20260927 02:53:03.307027 7213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:03.307253 7213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:03.308403 7213 hybrid_clock.cc:648] HybridClock initialized: now 1790477583308383 us; error 28 us; skew 500 ppm
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:03.310870 7213 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:53:03.311285 7213 webserver.cc:460] Webserver started at http://127.0.98.1:34169/ using document root <none> and password file <none>
I20260927 02:53:03.311496 7213 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:03.311563 7213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:03.313306 7213 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:03.313861 7318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.314016 7213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:03.314097 7213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "7a9748b30bac15fe5ba1f4f5990d8471"
server_key_iv: "cce28fc4efd66908a1070cf1a1e8fadf"
server_key_version: "encryptionkey@0"
I20260927 02:53:03.314436 7213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:03.336692 7213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:03.338629 7213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:03.338799 7213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:03.339020 7213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:03.339308 7213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:03.339344 7213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.339377 7213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:03.339406 7213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.345727 7213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41989
I20260927 02:53:03.345849 7431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41989 every 8 connection(s)
I20260927 02:53:03.346105 7213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:53:03.347102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 7213
I20260927 02:53:03.347242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 6663
I20260927 02:53:03.353272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43879
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34937
--webserver_port=32957
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457,127.0.98.60:40861 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:53:03.358167 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.346815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53253 (local address 127.0.98.60:40861)
0927 02:53:03.346919 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.346921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.347284 (+ 363us) server_negotiation.cc:407] Connection header received
0927 02:53:03.348400 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.348402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.348417 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.348459 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.349396 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.350436 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.351882 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.352195 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.352237 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.354931 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.354932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.354932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.354939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.356657 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.357121 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.357123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.357123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.357183 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.357460 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.357461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.357461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.357595 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.357641 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.357985 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.358067 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260927 02:53:03.358709 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.346841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.98.1:53253)
0927 02:53:03.347214 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.347224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.348284 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.348631 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.348636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.349017 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.349292 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.349298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.350529 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.350530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.351780 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.351782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.352339 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.352340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.352471 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.353020 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.353030 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.354837 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.356740 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.356747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.356749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.357035 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.357249 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.357250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.357250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.357384 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.357673 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.357679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.357832 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.358294 (+ 462us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.358478 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260927 02:53:03.358829 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.347732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44523 (local address 127.0.98.61:44457)
0927 02:53:03.347777 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.347778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.347833 (+ 55us) server_negotiation.cc:407] Connection header received
0927 02:53:03.350956 (+ 3123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.350958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.350972 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.351004 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.351423 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.352125 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.353005 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.353268 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.353316 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.356262 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.356263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.356263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.356270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.357631 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.358032 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.358033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.358033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.358085 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.358294 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.358295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.358295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.358419 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.358471 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.358651 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.358726 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:53:03.358903 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.346814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34367 (local address 127.0.98.62:34937)
0927 02:53:03.346915 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.346917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.350602 (+ 3685us) server_negotiation.cc:407] Connection header received
0927 02:53:03.350635 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.350636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.350650 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.350721 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.351093 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.351796 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.353011 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.353270 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.353310 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.355978 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.355980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.355980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.355988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.357526 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.357991 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.357992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.357992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.358057 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.358287 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.358288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.358288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.358426 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.358482 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.358682 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.358799 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 02:53:03.359071 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.347228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.98.1:44523)
0927 02:53:03.347782 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.347791 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.350857 (+ 3066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.351067 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.351068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.351245 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.351312 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.351314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.352216 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.352217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.352907 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.352910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.353634 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.353635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.353698 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.354016 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.354020 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.356171 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.357698 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.357699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.357699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.357962 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.358149 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.358150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.358150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.358226 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.358515 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.358515 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.358552 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.358902 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.358975 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":88,"threads_started":1}
I20260927 02:53:03.359191 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.346841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.1:34367)
0927 02:53:03.347026 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.347073 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.348276 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.350734 (+ 2458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.350735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.350927 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.351005 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.351007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.351894 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.351895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.352648 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.352650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.353388 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.353389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.353450 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.354007 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.354012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.355911 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.357621 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.357622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.357622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.357894 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.358133 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.358133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.358134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.358205 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.358517 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.358518 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.358562 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.358987 (+ 425us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.359085 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260927 02:53:03.359658 7432 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40861
I20260927 02:53:03.359665 7433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34937
I20260927 02:53:03.359743 7432 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.359754 7433 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.359890 7432 heartbeater.cc:511] Master 127.0.98.60:40861 requested a full tablet report, sending...
I20260927 02:53:03.359896 7433 heartbeater.cc:511] Master 127.0.98.62:34937 requested a full tablet report, sending...
I20260927 02:53:03.360288 6467 ts_manager.cc:194] Re-registered known tserver with Master: 9ddafcfe8d294d8f9ba5d2f3df2d7208 (127.0.98.1:41989)
I20260927 02:53:03.360339 7253 ts_manager.cc:194] Registered new tserver with Master: 9ddafcfe8d294d8f9ba5d2f3df2d7208 (127.0.98.1:41989)
I20260927 02:53:03.360697 6467 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34367
I20260927 02:53:03.361120 7434 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44457
I20260927 02:53:03.361166 7434 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.361234 7434 heartbeater.cc:511] Master 127.0.98.61:44457 requested a full tablet report, sending...
I20260927 02:53:03.361593 7065 ts_manager.cc:194] Registered new tserver with Master: 9ddafcfe8d294d8f9ba5d2f3df2d7208 (127.0.98.1:41989)
W20260927 02:53:03.437975 7440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:03.438177 7440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:03.438211 7440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:03.439401 7440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:03.439448 7440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:03.439484 7440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:03.440943 7440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32957
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457,127.0.98.60:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:03.441169 7440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:03.441385 7440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:03.446092 7448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.446130 7445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.446092 7446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:03.446349 7440 server_base.cc:1034] running on GCE node
I20260927 02:53:03.446545 7440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:03.446796 7440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:03.447950 7440 hybrid_clock.cc:648] HybridClock initialized: now 1790477583447934 us; error 31 us; skew 500 ppm
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:03.450332 7440 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:53:03.450780 7440 webserver.cc:460] Webserver started at http://127.0.98.2:32957/ using document root <none> and password file <none>
I20260927 02:53:03.450982 7440 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:03.451038 7440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:03.452277 7440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:03.452768 7455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.452898 7440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:03.452978 7440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "473f94ce017041b89d4ad07958552cd1"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "fa53ede76c4b4de3f63fa583ac07b301"
server_key_iv: "0796ffc15ec10c818451e0262d574ed4"
server_key_version: "encryptionkey@0"
I20260927 02:53:03.453299 7440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:03.463079 7440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:03.465119 7440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:03.465272 7440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:03.465488 7440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:03.465792 7440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:03.465828 7440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.465860 7440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:03.465885 7440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.471900 7440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43879
I20260927 02:53:03.471958 7568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43879 every 8 connection(s)
I20260927 02:53:03.472288 7440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:53:03.479543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 7440
I20260927 02:53:03.479684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 6797
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:53:03.482818 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.473083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48019 (local address 127.0.98.61:44457)
0927 02:53:03.473129 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.473131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.474231 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 02:53:03.474260 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.474261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.474271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.474316 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.474840 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.475341 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.476506 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.476687 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.476740 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.479651 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.479653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.479653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.479660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.481315 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.481759 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.481760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.481761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.481820 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.482088 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.482089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.482090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.482220 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.482260 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.482599 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.482683 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:53:03.483011 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.472999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.98.2:56421)
0927 02:53:03.473438 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.473462 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.473870 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.474360 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.474360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.474618 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.474731 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.474737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.476256 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.476257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.476903 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.476905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.477567 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.477568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.477631 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.477980 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.477985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.480032 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.481654 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.481655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.481655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.481866 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.482079 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.482080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.482080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.482138 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.482373 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.482374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.482463 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.482566 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.482841 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":141,"thread_start_us":77,"threads_started":1}
I20260927 02:53:03.483166 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.473335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56421 (local address 127.0.98.60:40861)
0927 02:53:03.473393 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.473394 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.473507 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:53:03.474240 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.474264 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.474279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.474342 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.475443 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.476155 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.476983 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.477256 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.477336 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.480134 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.480136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.480136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.480143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.481568 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.481966 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.481967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.481967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.482023 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.482193 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.482194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.482194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.482315 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.482375 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.482986 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.483075 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:53:03.483532 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.473000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.98.2:48019)
0927 02:53:03.474166 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.474174 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.474219 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.474319 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.474320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.474619 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.474731 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.474737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.475441 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.475442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.476427 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.476429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.476751 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.476752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.476881 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.477370 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.477380 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.479541 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.481397 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.481403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.481405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.481684 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.481890 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.481890 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.481891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.482012 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.482300 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.482305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.482468 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.482566 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.482748 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"mutex_wait_us":105,"thread_start_us":68,"threads_started":1}
I20260927 02:53:03.483933 7571 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44457
I20260927 02:53:03.484040 7571 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.484218 7571 heartbeater.cc:511] Master 127.0.98.61:44457 requested a full tablet report, sending...
I20260927 02:53:03.484647 7065 ts_manager.cc:194] Registered new tserver with Master: 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879)
I20260927 02:53:03.485751 7569 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40861
I20260927 02:53:03.485802 7569 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.485868 7569 heartbeater.cc:511] Master 127.0.98.60:40861 requested a full tablet report, sending...
I20260927 02:53:03.486128 7253 ts_manager.cc:194] Registered new tserver with Master: 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879)
I20260927 02:53:03.486233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39103
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34937
--webserver_port=36833
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457,127.0.98.60:40861 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:53:03.487782 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.473336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33491 (local address 127.0.98.62:34937)
0927 02:53:03.473441 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.473443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.473455 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:53:03.473947 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.473948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.473958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.474045 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.474892 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.475352 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.476795 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.476998 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.477060 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.479109 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.479111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.479111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.479117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.486713 (+ 7596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.487089 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.487090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.487091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.487150 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.487329 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.487329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.487330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.487439 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.487494 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.487623 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.487694 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 02:53:03.487923 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.472998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.2:33491)
0927 02:53:03.473204 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.473262 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.473856 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.474130 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.474134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.474536 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.474729 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.474734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.475444 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.475445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.476427 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.476429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.477112 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.477113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.477192 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.477580 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.477584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.479012 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.486841 (+ 7829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.486842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.486843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.487025 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.487201 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.487202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.487202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.487275 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.487492 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.487493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.487533 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.487594 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.487695 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":32,"thread_start_us":86,"threads_started":1}
I20260927 02:53:03.488298 7570 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34937
I20260927 02:53:03.488344 7570 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.488412 7570 heartbeater.cc:511] Master 127.0.98.62:34937 requested a full tablet report, sending...
I20260927 02:53:03.488636 6467 ts_manager.cc:194] Re-registered known tserver with Master: 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879)
I20260927 02:53:03.488997 6467 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33491
W20260927 02:53:03.565481 7576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:03.565681 7576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:03.565702 7576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:03.566864 7576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:03.566905 7576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:03.566931 7576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:03.568357 7576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36833
--tserver_master_addrs=127.0.98.62:34937,127.0.98.61:44457,127.0.98.60:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:03.568579 7576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:03.568816 7576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:03.573773 7582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.573776 7581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:03.573802 7584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:03.574301 7576 server_base.cc:1034] running on GCE node
I20260927 02:53:03.574499 7576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:03.574712 7576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:03.575865 7576 hybrid_clock.cc:648] HybridClock initialized: now 1790477583575851 us; error 29 us; skew 500 ppm
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:03.578215 7576 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:53:03.578595 7576 webserver.cc:460] Webserver started at http://127.0.98.3:36833/ using document root <none> and password file <none>
I20260927 02:53:03.578810 7576 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:03.578871 7576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:03.580171 7576 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:03.580690 7591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.580833 7576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:03.580909 7576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "4639e9d88baa4f69b7b025dbfe161727"
format_stamp: "Formatted at 2026-09-27 02:53:02 on dist-test-slave-mfx9"
server_key: "1eaf208be1f613c170c774222f1c8ce6"
server_key_iv: "750014ef9ab1d21085da9691a76367d0"
server_key_version: "encryptionkey@0"
I20260927 02:53:03.581375 7576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:03.602398 7576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:03.604844 7576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:03.604995 7576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:03.605228 7576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:03.605536 7576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:03.605572 7576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.605607 7576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:03.605634 7576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:03.611742 7576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39103
I20260927 02:53:03.611783 7704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39103 every 8 connection(s)
I20260927 02:53:03.612138 7576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:53:03 dist-test-slave-mfx9 krb5kdc[6422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:03.621658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 7576
I20260927 02:53:03.623826 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.612895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56891 (local address 127.0.98.61:44457)
0927 02:53:03.612937 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.612939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.613203 (+ 264us) server_negotiation.cc:407] Connection header received
0927 02:53:03.614188 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.614190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.614202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.614255 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.615285 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.615990 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617337 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.617628 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617678 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.620550 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.620551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.620552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.620560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.622172 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.622811 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.622813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.622813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.622871 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.623106 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.623107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.623107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.623234 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.623315 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.623659 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.623734 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:53:03.624397 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.613219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.3:54635)
0927 02:53:03.614239 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.614263 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.614336 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.614535 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.614536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.614932 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.615140 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.615149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.616530 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.616532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.617395 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.617398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.617853 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.617904 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.618424 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.618431 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.620983 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.623239 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.623239 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.623240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.623485 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.623713 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.623714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.623714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.623783 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.624074 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.624074 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.624113 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.624182 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.624262 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":28,"thread_start_us":56,"threads_started":1}
I20260927 02:53:03.624397 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.612915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.98.3:56891)
0927 02:53:03.613074 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.613108 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.614078 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.614349 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.614353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.614944 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.615141 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.615148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.616100 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.616101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.617204 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.617206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.617725 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.617811 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.618494 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.618498 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.620437 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.622274 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.622281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.622283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.622552 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.622935 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.622936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.622936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.623046 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.623344 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.623351 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.623505 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.623978 (+ 473us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.624147 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":8,"thread_start_us":93,"threads_started":1}
I20260927 02:53:03.624846 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.613306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54635 (local address 127.0.98.62:34937)
0927 02:53:03.613357 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.613358 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.614311 (+ 953us) server_negotiation.cc:407] Connection header received
0927 02:53:03.614395 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.614396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.614444 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.614506 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.615223 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.615994 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617467 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.617724 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617856 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.621098 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.621100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.621100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.621109 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.623157 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.623579 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.623580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.623580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.623643 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.623855 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.623856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.623857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.623983 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.624047 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.624667 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.624756 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:53:03.624861 7707 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44457
I20260927 02:53:03.624944 7707 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.625119 7707 heartbeater.cc:511] Master 127.0.98.61:44457 requested a full tablet report, sending...
I20260927 02:53:03.625237 7706 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34937
I20260927 02:53:03.625281 7706 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.625332 7706 heartbeater.cc:511] Master 127.0.98.62:34937 requested a full tablet report, sending...
I20260927 02:53:03.625530 7065 ts_manager.cc:194] Registered new tserver with Master: 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103)
I20260927 02:53:03.625584 6467 ts_manager.cc:194] Re-registered known tserver with Master: 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103)
I20260927 02:53:03.625917 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.621367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.0.1:57200)
0927 02:53:03.621419 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.621428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.621464 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.621706 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.621707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.621709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:03.621769 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.621771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.622660 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.622661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.623726 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.623729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.625699 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.625700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.625760 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.625764 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.625791 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.625850 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:53:03.626052 6467 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54635
I20260927 02:53:03.626084 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.612915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.98.3:53007)
0927 02:53:03.613368 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.613377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.614110 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.614348 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.614353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.614943 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:03.615141 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.615148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.616611 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.616612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.617200 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.617202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.617672 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.617807 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.618548 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:03.618554 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:03.620000 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:03.623246 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.623247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.623247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.625047 (+ 1800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.625253 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:03.625254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:03.625254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:03.625315 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:03.625595 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:03.625596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:03.625648 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.625704 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.625796 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":5,"thread_start_us":91,"threads_started":1}
I20260927 02:53:03.626259 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.621428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.0.98.60:40861)
0927 02:53:03.621548 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.621550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.621557 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:53:03.621578 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.621579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.621603 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.621835 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:03.621871 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.622542 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.623821 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.625616 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.626059 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.626141 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.626159 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.626193 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260927 02:53:03.626422 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.613360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53007 (local address 127.0.98.60:40861)
0927 02:53:03.613399 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.613401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.613411 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:53:03.614216 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.614218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.614232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.614293 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.616085 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.616539 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617296 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.617594 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.617652 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.620091 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.620093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.620093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.620099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.621941 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.625127 (+ 3186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.625128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.625128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.625186 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.625388 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.625388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.625389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.625509 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.625567 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.626236 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.626328 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:53:03.626457 7263 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 02:53:03.626636 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c01d122e7eb49a3a9b0f298da539414. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.626720 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:53:03.626717 7705 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40861
I20260927 02:53:03.626781 7705 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:03.626848 7705 heartbeater.cc:511] Master 127.0.98.60:40861 requested a full tablet report, sending...
I20260927 02:53:03.627089 7253 ts_manager.cc:194] Registered new tserver with Master: 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103)
I20260927 02:53:03.633929 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.624000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.0.98.60:40861)
0927 02:53:03.626333 (+ 2333us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.626335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.626341 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:53:03.626356 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.626356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.626365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.626408 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.627300 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.628000 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.630151 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.630327 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.630377 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.631516 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.631517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.631517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.631523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.632713 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.633054 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.633055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.633056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.633117 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.633305 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.633306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.633306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.633423 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.633510 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.633780 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.633849 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2321,"thread_start_us":40,"threads_started":1}
I20260927 02:53:03.634439 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.623782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.0.98.61:44457)
0927 02:53:03.623914 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.623916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.625981 (+ 2065us) server_negotiation.cc:407] Connection header received
0927 02:53:03.626186 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.626187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.626195 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.626220 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.628649 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.629084 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.629756 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.630044 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.630105 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.631770 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.631771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.631772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.631777 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.633142 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.633498 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.633499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.633499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.633556 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.633756 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.633757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.633757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.633875 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.633953 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.634303 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.634373 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":47,"threads_started":1}
I20260927 02:53:03.635031 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.624180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.0.98.62:34937)
0927 02:53:03.624660 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.624662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.625820 (+ 1158us) server_negotiation.cc:407] Connection header received
0927 02:53:03.625838 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.625839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.625851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.625908 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.627366 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.628063 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.629768 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.630052 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.630105 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.631997 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.631999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.631999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.632005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.633810 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.634195 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.634196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.634197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.634264 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.634479 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.634480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.634480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.634678 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.634790 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.634903 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.634962 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":67,"threads_started":1}
I20260927 02:53:03.636536 6467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:53:03.638087 6467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:53:03.638736 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: true } } }
I20260927 02:53:03.638864 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:03.640652 7027 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: attempting to promote NON_VOTER 30a21d18e77b47dc8d555a6f603c3807 to VOTER
I20260927 02:53:03.640753 7027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } }
I20260927 02:53:03.641230 7263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 LEARNER]: Refusing update from remote peer 4c01d122e7eb49a3a9b0f298da539414: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260927 02:53:03.641426 7074 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Refusing update from remote peer 4c01d122e7eb49a3a9b0f298da539414: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260927 02:53:03.641417 7026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:53:03.641587 7026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:53:03.642122 7224 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 30a21d18e77b47dc8d555a6f603c3807 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:03.642298 7263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 30a21d18e77b47dc8d555a6f603c3807 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:03.642338 7074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 30a21d18e77b47dc8d555a6f603c3807 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:03.643272 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:03.643352 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:03.643313 7217 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:03.643388 7217 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:53:03.643405 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:03.643450 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:03.643451 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:03.643478 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c01d122e7eb49a3a9b0f298da539414" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:03.643529 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:03.643527 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:53:03.647765 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.644254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41989 (local address 127.0.0.1:55400)
0927 02:53:03.644585 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.644595 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.644641 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.644794 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.644795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.644844 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:03.644889 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.644890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.645810 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.645811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.646596 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.646597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.647526 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.647527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.647587 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.647591 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.647649 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.647690 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":112,"threads_started":1}
I20260927 02:53:03.647782 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.644355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.0.98.1:41989)
0927 02:53:03.644626 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.644631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.644643 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:53:03.644707 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.644709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.644725 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.644793 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:03.644956 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.645734 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.646684 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.647440 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.647518 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.647591 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.647664 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.647705 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260927 02:53:03.648586 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.644225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39103 (local address 127.0.0.1:60188)
0927 02:53:03.644340 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.644350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.644385 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.645148 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.645149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.645150 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:03.645183 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.645185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.646437 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.646438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.647382 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.647384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.648360 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.648360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.648419 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.648423 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.648486 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.648521 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":47}
I20260927 02:53:03.648608 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.644477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.0.98.3:39103)
0927 02:53:03.645001 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.645005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.645015 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:53:03.645050 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.645052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.645065 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.645145 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:03.645242 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.646327 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.647461 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.648281 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.648352 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.648433 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.648500 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.648536 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":69,"threads_started":1}
I20260927 02:53:03.648643 7366 tablet_service.cc:1467] Processing CreateTablet for tablet a72753a7938146d5936cc069e268eac6 (DEFAULT_TABLE table=test-table [id=9049522583414e2eb77017793fed541d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:03.648922 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.644461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43879 (local address 127.0.0.1:53862)
0927 02:53:03.644807 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.644816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.644870 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.645118 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.645119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.645121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:03.645163 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.645165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.646419 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.646420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.647541 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.647544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.648452 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.648453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.648511 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.648515 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.648559 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.648596 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":22,"thread_start_us":61,"threads_started":1}
I20260927 02:53:03.648952 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.644566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.0.98.2:43879)
0927 02:53:03.644758 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.644761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.644879 (+ 118us) server_negotiation.cc:407] Connection header received
0927 02:53:03.644951 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.644953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.644970 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.645050 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:03.645255 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.646343 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.647626 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.648285 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.648357 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.648418 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.648636 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.648678 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260927 02:53:03.649011 7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72753a7938146d5936cc069e268eac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:03.649451 7639 tablet_service.cc:1467] Processing CreateTablet for tablet a72753a7938146d5936cc069e268eac6 (DEFAULT_TABLE table=test-table [id=9049522583414e2eb77017793fed541d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:03.649689 7503 tablet_service.cc:1467] Processing CreateTablet for tablet a72753a7938146d5936cc069e268eac6 (DEFAULT_TABLE table=test-table [id=9049522583414e2eb77017793fed541d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:03.650007 7503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72753a7938146d5936cc069e268eac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:03.650252 7639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72753a7938146d5936cc069e268eac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:03.653221 7735 tablet_bootstrap.cc:492] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Bootstrap starting.
I20260927 02:53:03.653798 7735 tablet_bootstrap.cc:654] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:03.654223 7735 log.cc:824] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:03.654366 7736 tablet_bootstrap.cc:492] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Bootstrap starting.
I20260927 02:53:03.654651 7738 tablet_bootstrap.cc:492] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: Bootstrap starting.
I20260927 02:53:03.654922 7736 tablet_bootstrap.cc:654] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:03.655149 7736 log.cc:824] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:03.655162 7738 tablet_bootstrap.cc:654] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:03.655386 7738 log.cc:824] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:03.655827 7736 tablet_bootstrap.cc:492] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: No bootstrap required, opened a new log
I20260927 02:53:03.655898 7736 ts_tablet_manager.cc:1397] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:53:03.656047 7738 tablet_bootstrap.cc:492] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: No bootstrap required, opened a new log
I20260927 02:53:03.656107 7738 ts_tablet_manager.cc:1397] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:53:03.657675 7736 raft_consensus.cc:348] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.657862 7736 raft_consensus.cc:374] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:03.657902 7736 raft_consensus.cc:729] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473f94ce017041b89d4ad07958552cd1, State: Initialized, Role: FOLLOWER
I20260927 02:53:03.658021 7736 consensus_queue.cc:260] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.658263 7736 ts_tablet_manager.cc:1428] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:53:03.658298 7570 heartbeater.cc:503] Master 127.0.98.62:34937 was elected leader, sending a full tablet report...
I20260927 02:53:03.658360 7738 raft_consensus.cc:348] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.658512 7738 raft_consensus.cc:374] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:03.658545 7738 raft_consensus.cc:729] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4639e9d88baa4f69b7b025dbfe161727, State: Initialized, Role: FOLLOWER
I20260927 02:53:03.658679 7735 tablet_bootstrap.cc:492] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: No bootstrap required, opened a new log
I20260927 02:53:03.658648 7738 consensus_queue.cc:260] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.658736 7735 ts_tablet_manager.cc:1397] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 02:53:03.658911 7738 ts_tablet_manager.cc:1428] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 02:53:03.659204 7706 heartbeater.cc:503] Master 127.0.98.62:34937 was elected leader, sending a full tablet report...
I20260927 02:53:03.661341 7735 raft_consensus.cc:348] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.661458 7735 raft_consensus.cc:374] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:03.661477 7735 raft_consensus.cc:729] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddafcfe8d294d8f9ba5d2f3df2d7208, State: Initialized, Role: FOLLOWER
I20260927 02:53:03.661577 7735 consensus_queue.cc:260] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.661793 7735 ts_tablet_manager.cc:1428] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:53:03.661832 7433 heartbeater.cc:503] Master 127.0.98.62:34937 was elected leader, sending a full tablet report...
I20260927 02:53:03.721084 7743 raft_consensus.cc:482] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:53:03.721196 7743 raft_consensus.cc:504] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.721590 7743 leader_election.cc:288] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103), 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879)
W20260927 02:53:03.723300 7572 tablet.cc:2369] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:53:03.724328 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.721963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57529 (local address 127.0.98.2:43879)
0927 02:53:03.722004 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.722006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.722013 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:53:03.722036 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.722037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.722046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.722089 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:03.722216 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.722680 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.723702 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.724045 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.724101 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.724111 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.724232 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.724268 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:53:03.724332 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.721757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43879 (local address 127.0.98.1:57529)
0927 02:53:03.721834 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.721846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.721902 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.722103 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.722103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.722105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:03.722160 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.722162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.722748 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.722749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.723618 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.723621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.724107 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.724108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.724178 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.724181 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.724223 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.724261 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":31}
I20260927 02:53:03.724545 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.721795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39103 (local address 127.0.98.1:41151)
0927 02:53:03.721857 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:03.721868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:03.721916 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:03.722049 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:03.722050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:03.722051 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:03.722122 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.722124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.722787 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.722788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.723749 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:03.723752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.724279 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.724280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:03.724355 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.724358 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:03.724411 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:53:03.724454 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:53:03.724674 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.721862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41151 (local address 127.0.98.3:39103)
0927 02:53:03.721935 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.721938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.721946 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:03.721973 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.721975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.721987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.722041 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:03.722181 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.722681 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.723839 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.724200 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.724266 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.724276 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.724569 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.724612 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:53:03.724671 7523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72753a7938146d5936cc069e268eac6" candidate_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473f94ce017041b89d4ad07958552cd1" is_pre_election: true
I20260927 02:53:03.724823 7523 raft_consensus.cc:2454] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddafcfe8d294d8f9ba5d2f3df2d7208 in term 0.
I20260927 02:53:03.724934 7659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72753a7938146d5936cc069e268eac6" candidate_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4639e9d88baa4f69b7b025dbfe161727" is_pre_election: true
I20260927 02:53:03.725067 7659 raft_consensus.cc:2454] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddafcfe8d294d8f9ba5d2f3df2d7208 in term 0.
I20260927 02:53:03.725068 7322 leader_election.cc:302] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 473f94ce017041b89d4ad07958552cd1, 9ddafcfe8d294d8f9ba5d2f3df2d7208; no voters:
I20260927 02:53:03.725199 7743 raft_consensus.cc:2781] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:53:03.725250 7743 raft_consensus.cc:482] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:53:03.725263 7743 raft_consensus.cc:3037] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:03.725925 7743 raft_consensus.cc:504] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.726053 7743 leader_election.cc:288] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [CANDIDATE]: Term 1 election: Requested vote from peers 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103), 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879)
I20260927 02:53:03.726217 7659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72753a7938146d5936cc069e268eac6" candidate_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4639e9d88baa4f69b7b025dbfe161727"
I20260927 02:53:03.726258 7523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72753a7938146d5936cc069e268eac6" candidate_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473f94ce017041b89d4ad07958552cd1"
I20260927 02:53:03.726310 7659 raft_consensus.cc:3037] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:03.726330 7523 raft_consensus.cc:3037] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:03.726974 7659 raft_consensus.cc:2454] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddafcfe8d294d8f9ba5d2f3df2d7208 in term 1.
I20260927 02:53:03.727124 7523 raft_consensus.cc:2454] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddafcfe8d294d8f9ba5d2f3df2d7208 in term 1.
I20260927 02:53:03.727139 7321 leader_election.cc:302] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4639e9d88baa4f69b7b025dbfe161727, 9ddafcfe8d294d8f9ba5d2f3df2d7208; no voters:
I20260927 02:53:03.727224 7743 raft_consensus.cc:2781] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:03.727375 7743 raft_consensus.cc:686] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 1 LEADER]: Becoming Leader. State: Replica: 9ddafcfe8d294d8f9ba5d2f3df2d7208, State: Running, Role: LEADER
I20260927 02:53:03.727453 7743 consensus_queue.cc:237] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } }
I20260927 02:53:03.729653 6467 catalog_manager.cc:5495] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ddafcfe8d294d8f9ba5d2f3df2d7208 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } health_report { overall_health: UNKNOWN } } }
I20260927 02:53:03.735929 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.733718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.0.98.1:41989)
0927 02:53:03.733755 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.733757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.733842 (+ 85us) server_negotiation.cc:407] Connection header received
0927 02:53:03.733866 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.733867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.733882 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.733934 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:03.734036 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.734463 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.735234 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.735424 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.735477 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.735598 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:03.735704 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:03.735763 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.735792 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.735865 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:53:03.737486 7523 raft_consensus.cc:1261] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 1 FOLLOWER]: Refusing update from remote peer 9ddafcfe8d294d8f9ba5d2f3df2d7208: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:03.737533 7659 raft_consensus.cc:1261] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 1 FOLLOWER]: Refusing update from remote peer 9ddafcfe8d294d8f9ba5d2f3df2d7208: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:03.737744 7743 consensus_queue.cc:1035] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [LEADER]: Connected to new peer: Peer: permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:03.737864 7744 consensus_queue.cc:1035] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:03.740899 7749 mvcc.cc:205] Tried to move back new op lower bound from 7333796182986678272 to 7333796182948474880. Current Snapshot: MvccSnapshot[applied={T|T < 7333796182986678272}]
I20260927 02:53:03.741492 7747 mvcc.cc:205] Tried to move back new op lower bound from 7333796182986678272 to 7333796182948474880. Current Snapshot: MvccSnapshot[applied={T|T < 7333796182986678272}]
I20260927 02:53:03.741597 7748 mvcc.cc:205] Tried to move back new op lower bound from 7333796182986678272 to 7333796182948474880. Current Snapshot: MvccSnapshot[applied={T|T < 7333796182986678272 or (T in {7333796182986678272})}]
I20260927 02:53:03.743939 6467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:03.744019 6467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:03.745114 6467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c01d122e7eb49a3a9b0f298da539414: Sending DeleteTablet for 3 replicas of tablet a72753a7938146d5936cc069e268eac6
I20260927 02:53:03.745461 7639 tablet_service.cc:1514] Processing DeleteTablet for tablet a72753a7938146d5936cc069e268eac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60188
I20260927 02:53:03.745563 7503 tablet_service.cc:1514] Processing DeleteTablet for tablet a72753a7938146d5936cc069e268eac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53862
I20260927 02:53:03.745695 7757 tablet_replica.cc:330] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: stopping tablet replica
I20260927 02:53:03.745810 7757 raft_consensus.cc:2229] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:03.745950 7757 raft_consensus.cc:2258] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:03.746279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 6435
I20260927 02:53:03.746335 7366 tablet_service.cc:1514] Processing DeleteTablet for tablet a72753a7938146d5936cc069e268eac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55400
I20260927 02:53:03.746693 7759 tablet_replica.cc:330] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: stopping tablet replica
I20260927 02:53:03.746438 7757 ts_tablet_manager.cc:1905] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:03.746961 7760 tablet_replica.cc:330] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: stopping tablet replica
I20260927 02:53:03.747085 7760 raft_consensus.cc:2229] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:53:03.747208 7759 raft_consensus.cc:2229] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:03.747372 7760 raft_consensus.cc:2258] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:03.747521 7759 raft_consensus.cc:2258] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:03.748016 7759 ts_tablet_manager.cc:1905] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:03.748337 7757 ts_tablet_manager.cc:1918] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:03.748405 7757 log.cc:1196] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/a72753a7938146d5936cc069e268eac6
I20260927 02:53:03.748585 7760 ts_tablet_manager.cc:1905] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:03.748656 7757 ts_tablet_manager.cc:1939] T a72753a7938146d5936cc069e268eac6 P 4639e9d88baa4f69b7b025dbfe161727: Deleting consensus metadata
I20260927 02:53:03.749989 7759 ts_tablet_manager.cc:1918] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:03.750058 7759 log.cc:1196] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/a72753a7938146d5936cc069e268eac6
I20260927 02:53:03.750311 7759 ts_tablet_manager.cc:1939] T a72753a7938146d5936cc069e268eac6 P 473f94ce017041b89d4ad07958552cd1: Deleting consensus metadata
I20260927 02:53:03.751987 7760 ts_tablet_manager.cc:1918] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:03.752056 7760 log.cc:1196] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/a72753a7938146d5936cc069e268eac6
I20260927 02:53:03.752333 7760 ts_tablet_manager.cc:1939] T a72753a7938146d5936cc069e268eac6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Deleting consensus metadata
W20260927 02:53:03.753104 7457 connection.cc:535] server connection from 127.0.0.1:53862 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 02:53:03.753118 7319 connection.cc:535] server connection from 127.0.0.1:55400 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 02:53:03.753305 7052 connection.cc:535] server connection from 127.0.0.1:56152 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 02:53:03.753437 7594 connection.cc:535] server connection from 127.0.0.1:60188 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:53:03.758247 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.745855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.0.98.60:40861)
0927 02:53:03.745894 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.745896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.749994 (+ 4098us) server_negotiation.cc:407] Connection header received
0927 02:53:03.750514 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.750515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.750527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.750564 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.752561 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.753117 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.754169 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.754422 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.754485 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.755853 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.755855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.755855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.755861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.757297 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.757588 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.757589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.757590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.757640 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.757797 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.757798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.757798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.757897 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.757975 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.758082 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.758158 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:53:03.761580 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.754823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.0.98.60:40861)
0927 02:53:03.755034 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.755036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.755044 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:03.755064 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.755064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.755075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.755147 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.755636 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.756411 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.757518 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.757827 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.757864 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.759278 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.759279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.759279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.759286 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.760553 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.760959 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.760960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.760960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.761005 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.761175 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.761176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.761176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.761283 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.761347 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.761469 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.761523 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260927 02:53:03.767977 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:03.763011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.0.98.61:44457)
0927 02:53:03.763048 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:03.763049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:03.763076 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:53:03.763095 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:03.763095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:03.763103 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:03.763144 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:03.763621 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.764086 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.764713 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:03.764895 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:03.764945 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:03.765892 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:03.765893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:03.765894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:03.765900 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:03.767088 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.767359 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.767359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.767360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.767410 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:03.767558 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:03.767559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:03.767560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:03.767656 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:03.767731 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:03.767831 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:53:03.767894 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:53:04.741423 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:04.741124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.3:49511)
0927 02:53:04.741268 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:04.741363 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
W20260927 02:53:04.741655 7706 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
I20260927 02:53:04.741967 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:04.741723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.2:45607)
0927 02:53:04.741849 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:04.741911 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
W20260927 02:53:04.742166 7570 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
I20260927 02:53:04.743196 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:04.742945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.1:50447)
0927 02:53:04.743097 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:04.743157 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":55,"threads_started":1}
W20260927 02:53:04.743372 7433 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
I20260927 02:53:05.245640 7770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c01d122e7eb49a3a9b0f298da539414)
I20260927 02:53:05.245641 7771 raft_consensus.cc:482] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c01d122e7eb49a3a9b0f298da539414)
I20260927 02:53:05.245757 7770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } }
I20260927 02:53:05.245775 7771 raft_consensus.cc:504] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } }
I20260927 02:53:05.246003 7770 leader_election.cc:288] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c01d122e7eb49a3a9b0f298da539414 (127.0.98.62:34937), b14b53aab66b4d2ba2bb21be1f0805d2 (127.0.98.61:44457)
I20260927 02:53:05.246047 7771 leader_election.cc:288] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c01d122e7eb49a3a9b0f298da539414 (127.0.98.62:34937), 30a21d18e77b47dc8d555a6f603c3807 (127.0.98.60:40861)
I20260927 02:53:05.246446 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.246124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:37428)
0927 02:53:05.246359 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.246406 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":51,"threads_started":1}
W20260927 02:53:05.246558 7240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
I20260927 02:53:05.246716 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.246260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:37434)
0927 02:53:05.246439 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.246674 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":52,"threads_started":1}
W20260927 02:53:05.246940 7052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
I20260927 02:53:05.246943 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.246870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:37436)
0927 02:53:05.246901 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.246921 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 02:53:05.247030 7240 leader_election.cc:334] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c01d122e7eb49a3a9b0f298da539414 (127.0.98.62:34937): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
I20260927 02:53:05.247460 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.247281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:37440)
0927 02:53:05.247398 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.247430 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
W20260927 02:53:05.247592 7052 leader_election.cc:334] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c01d122e7eb49a3a9b0f298da539414 (127.0.98.62:34937): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
I20260927 02:53:05.249343 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.246174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44457 (local address 127.0.0.1:56170)
0927 02:53:05.246434 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.246442 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:05.246522 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:05.246700 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:05.246701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:05.246703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:05.246748 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.246750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.247581 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.247582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.248353 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.248355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.249073 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.249074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.249144 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.249148 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:05.249200 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:53:05.249245 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":54,"threads_started":1}
I20260927 02:53:05.249348 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.246330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.0.98.61:44457)
0927 02:53:05.246402 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:05.246404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:05.246498 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:53:05.246586 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:05.246587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:05.246596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:05.246640 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:05.246847 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.247493 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.248435 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.248991 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.249071 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.249084 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:05.249212 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:53:05.249258 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"thread_start_us":40,"threads_started":1}
I20260927 02:53:05.249356 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.246260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.0.98.60:40861)
0927 02:53:05.246436 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:05.246438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:05.246448 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:53:05.246478 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:05.246479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:05.246490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:05.246744 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:05.246808 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.247501 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.248470 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.249010 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.249072 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.249084 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:05.249212 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:53:05.249261 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":29,"threads_started":1}
I20260927 02:53:05.249352 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.246148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40861 (local address 127.0.0.1:57240)
0927 02:53:05.246252 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.246263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:05.246316 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:05.246609 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:05.246611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:05.246613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:05.246676 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.246679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.247591 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.247593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.248366 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.248369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.249080 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.249081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.249144 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.249148 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:05.249200 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:53:05.249245 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":34,"threads_started":1}
I20260927 02:53:05.249604 7074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30a21d18e77b47dc8d555a6f603c3807" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" is_pre_election: true
I20260927 02:53:05.249704 7074 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a21d18e77b47dc8d555a6f603c3807 in term 1.
I20260927 02:53:05.249858 7238 leader_election.cc:302] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30a21d18e77b47dc8d555a6f603c3807, b14b53aab66b4d2ba2bb21be1f0805d2; no voters: 4c01d122e7eb49a3a9b0f298da539414
I20260927 02:53:05.249881 7263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30a21d18e77b47dc8d555a6f603c3807" is_pre_election: true
I20260927 02:53:05.249984 7770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:53:05.250041 7770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c01d122e7eb49a3a9b0f298da539414)
I20260927 02:53:05.250058 7770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:53:05.250725 7770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } }
I20260927 02:53:05.250864 7263 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b14b53aab66b4d2ba2bb21be1f0805d2 in current term 2: Already voted for candidate 30a21d18e77b47dc8d555a6f603c3807 in this term.
I20260927 02:53:05.250901 7770 leader_election.cc:288] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [CANDIDATE]: Term 2 election: Requested vote from peers 4c01d122e7eb49a3a9b0f298da539414 (127.0.98.62:34937), b14b53aab66b4d2ba2bb21be1f0805d2 (127.0.98.61:44457)
I20260927 02:53:05.251042 7051 leader_election.cc:302] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b14b53aab66b4d2ba2bb21be1f0805d2; no voters: 30a21d18e77b47dc8d555a6f603c3807, 4c01d122e7eb49a3a9b0f298da539414
I20260927 02:53:05.251075 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.250985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:37442)
0927 02:53:05.251031 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.251057 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:53:05.251210 7771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:53:05.251205 7074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30a21d18e77b47dc8d555a6f603c3807" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2"
I20260927 02:53:05.251376 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.251272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:37456)
0927 02:53:05.251334 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.251357 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 02:53:05.251439 7240 leader_election.cc:334] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4c01d122e7eb49a3a9b0f298da539414 (127.0.98.62:34937): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
I20260927 02:53:05.251894 7771 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:53:05.252446 7074 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a21d18e77b47dc8d555a6f603c3807 in term 2.
I20260927 02:53:05.252632 7238 leader_election.cc:302] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30a21d18e77b47dc8d555a6f603c3807, b14b53aab66b4d2ba2bb21be1f0805d2; no voters: 4c01d122e7eb49a3a9b0f298da539414
I20260927 02:53:05.252714 7770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:53:05.252847 7770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [term 2 LEADER]: Becoming Leader. State: Replica: 30a21d18e77b47dc8d555a6f603c3807, State: Running, Role: LEADER
I20260927 02:53:05.252913 7770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } }
I20260927 02:53:05.253204 7781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a21d18e77b47dc8d555a6f603c3807. Latest consensus state: current_term: 2 leader_uuid: "30a21d18e77b47dc8d555a6f603c3807" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:05.253278 7781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:05.253430 7783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:05.253798 7783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:53:05.253860 7783 catalog_manager.cc:1259] Loaded cluster ID: c5f4a13495f442cea6e805af7caf8b79
I20260927 02:53:05.253890 7783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:53:05.254045 7783 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:05.254112 7783 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Loaded TSK: 0
I20260927 02:53:05.254240 7783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:53:05.293009 7252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:53:05.294515 7252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:53:05.295464 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.295368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:37476)
0927 02:53:05.295402 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.295435 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:53:05.295486 7074 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [term 2 FOLLOWER]: Refusing update from remote peer 30a21d18e77b47dc8d555a6f603c3807: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 02:53:05.295712 7770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 02:53:05.295766 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.295695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.0.1:37482)
0927 02:53:05.295724 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.295751 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 02:53:05.295883 7240 consensus_peers.cc:487] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 -> Peer 4c01d122e7eb49a3a9b0f298da539414 (127.0.98.62:34937): Couldn't send request to peer 4c01d122e7eb49a3a9b0f298da539414. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:53:05.296399 7771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a21d18e77b47dc8d555a6f603c3807. Latest consensus state: current_term: 2 leader_uuid: "30a21d18e77b47dc8d555a6f603c3807" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:05.296497 7771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:53:05.296799 7784 mvcc.cc:205] Tried to move back new op lower bound from 7333796189368594432 to 7333796189196328960. Current Snapshot: MvccSnapshot[applied={T|T < 7333796189368594432}]
I20260927 02:53:05.296891 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "30a21d18e77b47dc8d555a6f603c3807" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:05.296962 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:05.296985 7771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "30a21d18e77b47dc8d555a6f603c3807" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4c01d122e7eb49a3a9b0f298da539414" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34937 } } peers { permanent_uuid: "b14b53aab66b4d2ba2bb21be1f0805d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44457 } attrs { promote: false } } peers { permanent_uuid: "30a21d18e77b47dc8d555a6f603c3807" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40861 } attrs { promote: false } } }
I20260927 02:53:05.297067 7771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14b53aab66b4d2ba2bb21be1f0805d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:53:05.297199 7785 mvcc.cc:205] Tried to move back new op lower bound from 7333796189368594432 to 7333796189196328960. Current Snapshot: MvccSnapshot[applied={T|T < 7333796189368594432 or (T in {7333796189368594432})}]
I20260927 02:53:05.301601 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.298651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39103 (local address 127.0.0.1:60204)
0927 02:53:05.298688 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.298697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:05.298730 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:05.298928 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:05.298929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:05.298931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:05.299008 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.299009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.299741 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.299742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.300555 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.300557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.301381 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.301382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.301437 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.301442 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:53:05.301493 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:53:05.301532 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:53:05.301604 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.298707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.0.98.3:39103)
0927 02:53:05.298824 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:05.298826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:05.298835 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:05.298859 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:05.298860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:05.298870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:05.298925 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:05.299079 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.299674 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.300644 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.301322 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.301378 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.301392 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:05.301502 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:53:05.301536 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260927 02:53:05.301878 7639 tablet_service.cc:1467] Processing CreateTablet for tablet 895923d7cfad4e70a8bc1210fa8c76a6 (DEFAULT_TABLE table=test-table [id=24e3dfc90c3c4257834a664d67f52449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:05.301916 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.298817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.0.98.1:41989)
0927 02:53:05.298901 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:05.298903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:05.299052 (+ 149us) server_negotiation.cc:407] Connection header received
0927 02:53:05.299167 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:05.299167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:05.299179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:05.299213 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:05.299378 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.300071 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.301020 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.301559 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.301618 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.301629 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:05.301803 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 02:53:05.301847 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":39,"threads_started":1}
I20260927 02:53:05.302004 7639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895923d7cfad4e70a8bc1210fa8c76a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:05.302103 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.298662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43879 (local address 127.0.0.1:53868)
0927 02:53:05.298857 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.298865 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:05.298904 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:05.299096 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:05.299097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:05.299099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:05.299141 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.299143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.300071 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.300072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.301195 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.301197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.301893 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.301894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.301952 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.301955 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:05.301982 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 02:53:05.302021 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":55}
I20260927 02:53:05.302121 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.298732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41989 (local address 127.0.0.1:55416)
0927 02:53:05.298999 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.299008 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:05.299094 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:05.299271 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:05.299272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:05.299273 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:05.299311 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.299312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.300149 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.300149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.300936 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:05.300938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.301658 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.301659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:05.301717 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.301721 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:05.302020 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 02:53:05.302054 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260927 02:53:05.302352 7366 tablet_service.cc:1467] Processing CreateTablet for tablet 895923d7cfad4e70a8bc1210fa8c76a6 (DEFAULT_TABLE table=test-table [id=24e3dfc90c3c4257834a664d67f52449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:05.302431 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.298821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.0.98.2:43879)
0927 02:53:05.298951 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:05.298953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:05.298961 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:05.298990 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:05.298991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:05.299001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:05.299198 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:05.299235 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.299985 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.301265 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.301819 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.302301 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.302316 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:05.302333 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 02:53:05.302370 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
I20260927 02:53:05.302462 7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895923d7cfad4e70a8bc1210fa8c76a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:05.302603 7503 tablet_service.cc:1467] Processing CreateTablet for tablet 895923d7cfad4e70a8bc1210fa8c76a6 (DEFAULT_TABLE table=test-table [id=24e3dfc90c3c4257834a664d67f52449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:05.302701 7503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895923d7cfad4e70a8bc1210fa8c76a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:05.303468 7793 tablet_bootstrap.cc:492] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: Bootstrap starting.
I20260927 02:53:05.303763 7793 tablet_bootstrap.cc:654] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:05.303871 7794 tablet_bootstrap.cc:492] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Bootstrap starting.
I20260927 02:53:05.304075 7796 tablet_bootstrap.cc:492] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: Bootstrap starting.
I20260927 02:53:05.304236 7794 tablet_bootstrap.cc:654] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:05.304522 7793 tablet_bootstrap.cc:492] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: No bootstrap required, opened a new log
I20260927 02:53:05.304567 7793 ts_tablet_manager.cc:1397] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:05.304699 7794 tablet_bootstrap.cc:492] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: No bootstrap required, opened a new log
I20260927 02:53:05.304744 7794 ts_tablet_manager.cc:1397] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:05.304714 7793 raft_consensus.cc:348] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.304780 7793 raft_consensus.cc:374] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:05.304807 7793 raft_consensus.cc:729] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4639e9d88baa4f69b7b025dbfe161727, State: Initialized, Role: FOLLOWER
I20260927 02:53:05.304838 7794 raft_consensus.cc:348] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.304869 7793 consensus_queue.cc:260] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.304898 7794 raft_consensus.cc:374] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:05.304919 7794 raft_consensus.cc:729] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddafcfe8d294d8f9ba5d2f3df2d7208, State: Initialized, Role: FOLLOWER
I20260927 02:53:05.304955 7794 consensus_queue.cc:260] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.305018 7793 ts_tablet_manager.cc:1428] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:05.305074 7794 ts_tablet_manager.cc:1428] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:05.305399 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.305206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.3:38981)
0927 02:53:05.305325 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.305360 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260927 02:53:05.305550 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.305260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.1:53985)
0927 02:53:05.305489 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.305521 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260927 02:53:05.304371 7796 tablet_bootstrap.cc:654] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:05.306005 7705 heartbeater.cc:503] Master 127.0.98.60:40861 was elected leader, sending a full tablet report...
I20260927 02:53:05.306141 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.305962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.3:52137)
0927 02:53:05.306091 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.306120 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 02:53:05.306584 7796 tablet_bootstrap.cc:492] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: No bootstrap required, opened a new log
I20260927 02:53:05.306624 7796 ts_tablet_manager.cc:1397] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:53:05.306701 7796 raft_consensus.cc:348] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.306749 7796 raft_consensus.cc:374] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:05.306792 7796 raft_consensus.cc:729] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473f94ce017041b89d4ad07958552cd1, State: Initialized, Role: FOLLOWER
I20260927 02:53:05.306829 7796 consensus_queue.cc:260] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.307018 7796 ts_tablet_manager.cc:1428] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:05.307399 7569 heartbeater.cc:503] Master 127.0.98.60:40861 was elected leader, sending a full tablet report...
I20260927 02:53:05.307468 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.307156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.2:36491)
0927 02:53:05.307262 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.307403 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":75,"threads_started":1}
I20260927 02:53:05.307797 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.307647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.2:37585)
0927 02:53:05.307753 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.307775 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
W20260927 02:53:05.348460 7435 tablet.cc:2369] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:53:05.363528 7708 tablet.cc:2369] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:53:05.458240 7799 raft_consensus.cc:482] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:53:05.458349 7799 raft_consensus.cc:504] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.458531 7799 leader_election.cc:288] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103), 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879)
I20260927 02:53:05.458745 7659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895923d7cfad4e70a8bc1210fa8c76a6" candidate_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4639e9d88baa4f69b7b025dbfe161727" is_pre_election: true
I20260927 02:53:05.458745 7523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895923d7cfad4e70a8bc1210fa8c76a6" candidate_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473f94ce017041b89d4ad07958552cd1" is_pre_election: true
I20260927 02:53:05.458864 7523 raft_consensus.cc:2454] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddafcfe8d294d8f9ba5d2f3df2d7208 in term 0.
I20260927 02:53:05.458863 7659 raft_consensus.cc:2454] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddafcfe8d294d8f9ba5d2f3df2d7208 in term 0.
I20260927 02:53:05.459023 7321 leader_election.cc:302] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4639e9d88baa4f69b7b025dbfe161727, 9ddafcfe8d294d8f9ba5d2f3df2d7208; no voters:
I20260927 02:53:05.459127 7799 raft_consensus.cc:2781] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:53:05.459167 7799 raft_consensus.cc:482] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:53:05.459179 7799 raft_consensus.cc:3037] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:05.459829 7799 raft_consensus.cc:504] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.459941 7799 leader_election.cc:288] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [CANDIDATE]: Term 1 election: Requested vote from peers 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103), 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879)
I20260927 02:53:05.460106 7523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895923d7cfad4e70a8bc1210fa8c76a6" candidate_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473f94ce017041b89d4ad07958552cd1"
I20260927 02:53:05.460172 7523 raft_consensus.cc:3037] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:05.460135 7659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895923d7cfad4e70a8bc1210fa8c76a6" candidate_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4639e9d88baa4f69b7b025dbfe161727"
I20260927 02:53:05.460197 7659 raft_consensus.cc:3037] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:05.460747 7523 raft_consensus.cc:2454] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddafcfe8d294d8f9ba5d2f3df2d7208 in term 1.
I20260927 02:53:05.460839 7659 raft_consensus.cc:2454] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddafcfe8d294d8f9ba5d2f3df2d7208 in term 1.
I20260927 02:53:05.460901 7322 leader_election.cc:302] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 473f94ce017041b89d4ad07958552cd1, 9ddafcfe8d294d8f9ba5d2f3df2d7208; no voters:
I20260927 02:53:05.460975 7799 raft_consensus.cc:2781] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:05.461110 7799 raft_consensus.cc:686] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [term 1 LEADER]: Becoming Leader. State: Replica: 9ddafcfe8d294d8f9ba5d2f3df2d7208, State: Running, Role: LEADER
I20260927 02:53:05.461158 7799 consensus_queue.cc:237] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } }
I20260927 02:53:05.461181 7432 heartbeater.cc:503] Master 127.0.98.60:40861 was elected leader, sending a full tablet report...
I20260927 02:53:05.461748 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.461640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.1:47399)
0927 02:53:05.461688 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.461717 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:53:05.461717 7252 catalog_manager.cc:5495] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ddafcfe8d294d8f9ba5d2f3df2d7208 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ddafcfe8d294d8f9ba5d2f3df2d7208" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41989 } health_report { overall_health: HEALTHY } } }
I20260927 02:53:05.480909 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.479071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.0.98.1:41989)
0927 02:53:05.479109 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:05.479111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:05.479119 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:05.479160 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:05.479161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:05.479171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:05.479203 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:05.479289 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.479740 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.480407 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:05.480594 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:05.480636 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:05.480719 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:05.480743 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:05.480803 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:05.480836 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 02:53:05.480854 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:53:05.481683 7523 raft_consensus.cc:1261] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 1 FOLLOWER]: Refusing update from remote peer 9ddafcfe8d294d8f9ba5d2f3df2d7208: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:05.481716 7659 raft_consensus.cc:1261] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 1 FOLLOWER]: Refusing update from remote peer 9ddafcfe8d294d8f9ba5d2f3df2d7208: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:05.481911 7806 consensus_queue.cc:1035] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [LEADER]: Connected to new peer: Peer: permanent_uuid: "473f94ce017041b89d4ad07958552cd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:05.482016 7799 consensus_queue.cc:1035] T 895923d7cfad4e70a8bc1210fa8c76a6 P 9ddafcfe8d294d8f9ba5d2f3df2d7208 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4639e9d88baa4f69b7b025dbfe161727" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:05.483153 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.483052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.2:37285)
0927 02:53:05.483090 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.483122 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 02:53:05.483173 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.482917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.3:54411)
0927 02:53:05.483106 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.483141 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260927 02:53:05.483637 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.483550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.1:50331)
0927 02:53:05.483582 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.483608 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 02:53:05.484238 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.484147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.3:60669)
0927 02:53:05.484185 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.484213 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 02:53:05.485056 7252 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:05.485143 7252 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:05.485860 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.485758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.2:34195)
0927 02:53:05.485799 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.485831 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:53:05.486044 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.484952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34937 (local address 127.0.98.1:60065)
0927 02:53:05.485986 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.486015 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1017}
I20260927 02:53:05.486469 7252 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30a21d18e77b47dc8d555a6f603c3807: Sending DeleteTablet for 3 replicas of tablet 895923d7cfad4e70a8bc1210fa8c76a6
I20260927 02:53:05.486639 7503 tablet_service.cc:1514] Processing DeleteTablet for tablet 895923d7cfad4e70a8bc1210fa8c76a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53868
I20260927 02:53:05.486969 7639 tablet_service.cc:1514] Processing DeleteTablet for tablet 895923d7cfad4e70a8bc1210fa8c76a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60204
I20260927 02:53:05.486996 7503 tablet_service.cc:1514] Processing DeleteTablet for tablet 895923d7cfad4e70a8bc1210fa8c76a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53868
I20260927 02:53:05.487098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 7213
I20260927 02:53:05.487131 7819 tablet_replica.cc:330] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: stopping tablet replica
I20260927 02:53:05.488039 7819 raft_consensus.cc:2229] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:05.488098 7819 raft_consensus.cc:2258] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:05.488348 7819 ts_tablet_manager.cc:1905] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:05.489668 7819 ts_tablet_manager.cc:1918] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:05.489718 7819 log.cc:1196] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/895923d7cfad4e70a8bc1210fa8c76a6
I20260927 02:53:05.489926 7819 ts_tablet_manager.cc:1939] T 895923d7cfad4e70a8bc1210fa8c76a6 P 473f94ce017041b89d4ad07958552cd1: Deleting consensus metadata
I20260927 02:53:05.490258 7240 catalog_manager.cc:4841] TS 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879): tablet 895923d7cfad4e70a8bc1210fa8c76a6 (table test-table [id=24e3dfc90c3c4257834a664d67f52449]) successfully deleted
W20260927 02:53:05.490419 7240 catalog_manager.cc:4808] TS 473f94ce017041b89d4ad07958552cd1 (127.0.98.2:43879): delete failed for tablet 895923d7cfad4e70a8bc1210fa8c76a6 because the tablet was not found. No further retry: Not found: Tablet not found: 895923d7cfad4e70a8bc1210fa8c76a6
I20260927 02:53:05.493232 7820 tablet_replica.cc:330] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: stopping tablet replica
I20260927 02:53:05.493306 7820 raft_consensus.cc:2229] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:05.493355 7820 raft_consensus.cc:2258] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:05.493345 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:05.493239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41989 (local address 127.0.0.1:55434)
0927 02:53:05.493286 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:05.493312 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:53:05.493460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 7440
W20260927 02:53:05.493494 7240 catalog_manager.cc:4584] TS 9ddafcfe8d294d8f9ba5d2f3df2d7208 (127.0.98.1:41989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 895923d7cfad4e70a8bc1210fa8c76a6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41989: connect: Connection refused (error 111)
I20260927 02:53:05.493686 7820 ts_tablet_manager.cc:1905] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:05.495249 7820 ts_tablet_manager.cc:1918] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:05.495307 7820 log.cc:1196] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/895923d7cfad4e70a8bc1210fa8c76a6
I20260927 02:53:05.495549 7820 ts_tablet_manager.cc:1939] T 895923d7cfad4e70a8bc1210fa8c76a6 P 4639e9d88baa4f69b7b025dbfe161727: Deleting consensus metadata
I20260927 02:53:05.495839 7239 catalog_manager.cc:4841] TS 4639e9d88baa4f69b7b025dbfe161727 (127.0.98.3:39103): tablet 895923d7cfad4e70a8bc1210fa8c76a6 (table test-table [id=24e3dfc90c3c4257834a664d67f52449]) successfully deleted
I20260927 02:53:05.497874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 7576
I20260927 02:53:05.502074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 6931
I20260927 02:53:05.505890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 7119
2026-09-27T02:53:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4446 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:53:05 dist-test-slave-mfx9 krb5kdc[7823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:53:05 dist-test-slave-mfx9 krb5kdc[7823](info): set up 2 sockets
Sep 27 02:53:05 dist-test-slave-mfx9 krb5kdc[7823](info): commencing operation
krb5kdc: starting...
W20260927 02:53:06.450819 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:53:06 dist-test-slave-mfx9 krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:53:06.559393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:06.635308 7836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:06.635488 7836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:06.635510 7836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:06.636606 7836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:06.636643 7836 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:53:06.636655 7836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:06.636667 7836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:06.636677 7836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:06.636690 7836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:06.638075 7836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43331
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:06.638262 7836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:06.638468 7836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:06.642838 7842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:06.642966 7836 server_base.cc:1034] running on GCE node
W20260927 02:53:06.642920 7841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:06.642899 7844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:06.643219 7836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:06.643460 7836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:06.644598 7836 hybrid_clock.cc:648] HybridClock initialized: now 1790477586644583 us; error 26 us; skew 500 ppm
Sep 27 02:53:06 dist-test-slave-mfx9 krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477586, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:06.651023 7836 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:53:06.651403 7836 webserver.cc:460] Webserver started at http://127.0.98.62:43579/ using document root <none> and password file <none>
I20260927 02:53:06.651597 7836 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:06.651641 7836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:06.651757 7836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:06.652705 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "6e23abef7edc441da1be047ac15925a9"
format_stamp: "Formatted at 2026-09-27 02:53:06 on dist-test-slave-mfx9"
server_key: "a3ee4bebc3277911cb157039a46abdf0"
server_key_iv: "e081c72f33fdf9a39f5a39472644b0d8"
server_key_version: "encryptionkey@0"
I20260927 02:53:06.653029 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e23abef7edc441da1be047ac15925a9"
format_stamp: "Formatted at 2026-09-27 02:53:06 on dist-test-slave-mfx9"
server_key: "a3ee4bebc3277911cb157039a46abdf0"
server_key_iv: "e081c72f33fdf9a39f5a39472644b0d8"
server_key_version: "encryptionkey@0"
I20260927 02:53:06.654134 7836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:53:06.654731 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:06.654909 7836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:06.654986 7836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "6e23abef7edc441da1be047ac15925a9"
format_stamp: "Formatted at 2026-09-27 02:53:06 on dist-test-slave-mfx9"
server_key: "a3ee4bebc3277911cb157039a46abdf0"
server_key_iv: "e081c72f33fdf9a39f5a39472644b0d8"
server_key_version: "encryptionkey@0"
I20260927 02:53:06.655045 7836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:06.681619 7836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:06.683442 7836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:06.683575 7836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:06.687780 7836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43331
I20260927 02:53:06.687884 7913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43331 every 8 connection(s)
I20260927 02:53:06.688148 7836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:53:06.688700 7914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:06.690937 7914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: Bootstrap starting.
I20260927 02:53:06.691475 7914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:06.691721 7914 log.cc:824] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:06.692296 7914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: No bootstrap required, opened a new log
I20260927 02:53:06.693585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 7836
I20260927 02:53:06.693677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:53:06.693872 392 external_mini_cluster.cc:1428] Setting key 89c461c1e90d533be13f5a138e4097da
I20260927 02:53:06.694334 7914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23abef7edc441da1be047ac15925a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43331 } }
I20260927 02:53:06.694514 7914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:06.694551 7914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e23abef7edc441da1be047ac15925a9, State: Initialized, Role: FOLLOWER
I20260927 02:53:06.694671 7914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23abef7edc441da1be047ac15925a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43331 } }
I20260927 02:53:06.694815 7914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:53:06.694859 7914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:53:06.694905 7914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:06.695394 7914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23abef7edc441da1be047ac15925a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43331 } }
I20260927 02:53:06.695485 7914 leader_election.cc:302] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e23abef7edc441da1be047ac15925a9; no voters:
I20260927 02:53:06.695623 7914 leader_election.cc:288] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:53:06.695673 7919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:06.695811 7919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6e23abef7edc441da1be047ac15925a9, State: Running, Role: LEADER
I20260927 02:53:06.695848 7914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:06.695899 7919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23abef7edc441da1be047ac15925a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43331 } }
I20260927 02:53:06.696210 7919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e23abef7edc441da1be047ac15925a9. Latest consensus state: current_term: 1 leader_uuid: "6e23abef7edc441da1be047ac15925a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23abef7edc441da1be047ac15925a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43331 } } }
I20260927 02:53:06.696280 7919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:06.696219 7921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e23abef7edc441da1be047ac15925a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23abef7edc441da1be047ac15925a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43331 } } }
I20260927 02:53:06.696313 7921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:06.696627 7926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:06.697079 7926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:53:06.699198 7926 catalog_manager.cc:1347] Generated new cluster ID: d0d60f3f03e84abdacd07998a0591209
I20260927 02:53:06.699249 7926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:53:06 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:06.704610 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:06.694358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.0.98.62:43331)
0927 02:53:06.694529 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:06.694536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:06.694566 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:53:06.694696 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:06.694702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:06.694743 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:06.694905 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:06.695511 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.696878 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.697949 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.698523 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.698619 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:06.701167 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:06.701171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:06.701172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:06.701183 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:06.703082 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:06.703426 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:06.703429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:06.703430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:06.703548 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:06.703719 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:06.703719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:06.703720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:06.704035 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:06.704202 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:06.704269 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:53:06.704416 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260927 02:53:06.711158 7926 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:53:06.711630 7926 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:06.716432 7926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: Generated new TSK 0
I20260927 02:53:06.716583 7926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:53:06.742982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43331
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:06.843809 7940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:06.843991 7940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:06.844013 7940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:06.845149 7940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:06.845185 7940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:06.845204 7940 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:53:06.845234 7940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:06.846661 7940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:06.846910 7940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:06.847131 7940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:06.851728 7946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:06.851737 7948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:06.851743 7945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:06.851938 7940 server_base.cc:1034] running on GCE node
I20260927 02:53:06.852171 7940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:06.852384 7940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:06.853524 7940 hybrid_clock.cc:648] HybridClock initialized: now 1790477586853509 us; error 30 us; skew 500 ppm
Sep 27 02:53:06 dist-test-slave-mfx9 krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477586, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:06.855952 7940 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:53:06.856338 7940 webserver.cc:460] Webserver started at http://127.0.98.1:45305/ using document root <none> and password file <none>
I20260927 02:53:06.856514 7940 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:06.856550 7940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:06.856634 7940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:06.857646 7940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "34282bd6edb14a04a35651b4b019dafd"
format_stamp: "Formatted at 2026-09-27 02:53:06 on dist-test-slave-mfx9"
server_key: "b1c068d77d04979e7d35d36f5e4ffbda"
server_key_iv: "401c36c9ec894e4f2b96b67eff6c06e4"
server_key_version: "encryptionkey@0"
I20260927 02:53:06.857939 7940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "34282bd6edb14a04a35651b4b019dafd"
format_stamp: "Formatted at 2026-09-27 02:53:06 on dist-test-slave-mfx9"
server_key: "b1c068d77d04979e7d35d36f5e4ffbda"
server_key_iv: "401c36c9ec894e4f2b96b67eff6c06e4"
server_key_version: "encryptionkey@0"
I20260927 02:53:06.859138 7940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:53:06.859766 7955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:06.859961 7940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:53:06.860040 7940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "34282bd6edb14a04a35651b4b019dafd"
format_stamp: "Formatted at 2026-09-27 02:53:06 on dist-test-slave-mfx9"
server_key: "b1c068d77d04979e7d35d36f5e4ffbda"
server_key_iv: "401c36c9ec894e4f2b96b67eff6c06e4"
server_key_version: "encryptionkey@0"
I20260927 02:53:06.860090 7940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:06.890401 7940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:06.892514 7940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:06.892624 7940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:06.893062 7940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:06.893100 7940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:06.893126 7940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:06.893141 7940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:53:06 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477586, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:06.899732 7940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34077
I20260927 02:53:06.899848 8070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34077 every 8 connection(s)
I20260927 02:53:06.900141 7940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:53:06.902731 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:06.893570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55151 (local address 127.0.98.62:43331)
0927 02:53:06.893617 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:06.893620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:06.893871 (+ 251us) server_negotiation.cc:407] Connection header received
0927 02:53:06.894353 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:06.894355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:06.894369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:06.894443 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:06.895240 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.896019 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.897355 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.897691 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.897769 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:06.900075 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:06.900077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:06.900077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:06.900086 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:06.901452 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:06.901801 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:06.901802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:06.901803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:06.901869 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:06.902076 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:06.902076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:06.902077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:06.902187 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:06.902240 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:06.902505 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 02:53:06.902615 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:53:06.902869 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:06.893572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43331 (local address 127.0.98.1:55151)
0927 02:53:06.893711 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:06.893744 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:06.894239 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:06.894507 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:06.894511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:06.894860 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:06.895130 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:06.895136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.896103 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.896104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:06.897270 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:06.897273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.897788 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.897789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:06.897930 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:06.898400 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:06.898408 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:06.899980 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:06.901525 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:06.901531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:06.901533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:06.901737 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:06.901922 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:06.901922 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:06.901923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:06.902025 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:06.902237 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:06.902244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:06.902377 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:53:06.902466 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:53:06.902619 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260927 02:53:06.903309 8071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43331
I20260927 02:53:06.903389 8071 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:06.903551 8071 heartbeater.cc:511] Master 127.0.98.62:43331 requested a full tablet report, sending...
I20260927 02:53:06.903997 7867 ts_manager.cc:194] Registered new tserver with Master: 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077)
I20260927 02:53:06.904743 7867 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55151
I20260927 02:53:06.907851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 7940
I20260927 02:53:06.907949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:53:06.908138 392 external_mini_cluster.cc:1428] Setting key 9bea42fd572ebdb4571ff9457465d1f0
I20260927 02:53:06.909186 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:06.904173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43331 (local address 127.0.98.1:45977)
0927 02:53:06.904318 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:06.904354 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:06.904400 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:06.904502 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:06.904503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:06.904631 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:06.904683 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:06.904684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.905407 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.905408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:06.905934 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:06.905936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.906247 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.906248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:06.906307 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:06.906682 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:06.906686 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:06.907016 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:06.908286 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:06.908287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:06.908287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:06.908475 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:06.908631 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:06.908631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:06.908632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:06.908689 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:06.908945 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:06.908945 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:06.909000 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 02:53:06.909050 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:53:06.909120 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260927 02:53:06.909235 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:06.904240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45977 (local address 127.0.98.62:43331)
0927 02:53:06.904282 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:06.904284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:06.904405 (+ 121us) server_negotiation.cc:407] Connection header received
0927 02:53:06.904430 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:06.904431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:06.904440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:06.904498 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:06.904732 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.905307 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.905995 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:06.906168 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:06.906213 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:06.907072 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:06.907072 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:06.907073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:06.907077 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:06.908216 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:06.908539 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:06.908540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:06.908540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:06.908584 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:06.908738 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:06.908739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:06.908739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:06.908885 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:06.908945 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:06.909084 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:53:06.909155 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:53:06.929140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43331
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:07.005982 8080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:07.006186 8080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:07.006209 8080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:07.007364 8080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:07.007403 8080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:07.007422 8080 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:53:07.007438 8080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:07.008908 8080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:07.009140 8080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:07.009377 8080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:07.014165 8088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:07.014315 8080 server_base.cc:1034] running on GCE node
W20260927 02:53:07.014189 8085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:07.014386 8086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:07.014600 8080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:07.014837 8080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:07.015959 8080 hybrid_clock.cc:648] HybridClock initialized: now 1790477587015954 us; error 31 us; skew 500 ppm
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477587, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:07.018324 8080 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:53:07.018834 8080 webserver.cc:460] Webserver started at http://127.0.98.2:42037/ using document root <none> and password file <none>
I20260927 02:53:07.019048 8080 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:07.019090 8080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:07.019203 8080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:07.020210 8080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "c84e02f9699042f6b406d52d0fc3e664"
format_stamp: "Formatted at 2026-09-27 02:53:07 on dist-test-slave-mfx9"
server_key: "6d4e0112c8fe73caa1624350ab1c343b"
server_key_iv: "dea7e5e4222731ec9895ff937c8d0f83"
server_key_version: "encryptionkey@0"
I20260927 02:53:07.020538 8080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c84e02f9699042f6b406d52d0fc3e664"
format_stamp: "Formatted at 2026-09-27 02:53:07 on dist-test-slave-mfx9"
server_key: "6d4e0112c8fe73caa1624350ab1c343b"
server_key_iv: "dea7e5e4222731ec9895ff937c8d0f83"
server_key_version: "encryptionkey@0"
I20260927 02:53:07.021728 8080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:53:07.022370 8095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.022509 8080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:53:07.022589 8080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "c84e02f9699042f6b406d52d0fc3e664"
format_stamp: "Formatted at 2026-09-27 02:53:07 on dist-test-slave-mfx9"
server_key: "6d4e0112c8fe73caa1624350ab1c343b"
server_key_iv: "dea7e5e4222731ec9895ff937c8d0f83"
server_key_version: "encryptionkey@0"
I20260927 02:53:07.022653 8080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:07.034874 8080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:07.036890 8080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:07.037022 8080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:07.037498 8080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:07.037539 8080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.037568 8080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:07.037585 8080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.043866 8080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44439
I20260927 02:53:07.043949 8210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44439 every 8 connection(s)
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477587, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:07.044245 8080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:53:07.047155 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.037932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49747 (local address 127.0.98.62:43331)
0927 02:53:07.038002 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.038005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.038198 (+ 193us) server_negotiation.cc:407] Connection header received
0927 02:53:07.038678 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.038679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.038692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.038808 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.039570 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.040315 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.041879 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.042162 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.042232 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.044658 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:07.044660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:07.044661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:07.044669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:07.045823 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.046192 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.046193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.046194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.046256 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.046554 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.046555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.046555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.046673 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:07.046735 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.047014 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.047088 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:53:07.047266 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.037930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43331 (local address 127.0.98.2:49747)
0927 02:53:07.038075 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.038116 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.038585 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.038865 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.038869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.039172 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:07.039463 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.039470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.040405 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.040406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.041776 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.041780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.042251 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.042252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.042360 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.042932 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:07.042941 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:07.044540 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:07.045896 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.045901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.045902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.046111 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.046352 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.046353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.046353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.046476 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.046727 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:07.046730 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:07.046887 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.046974 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.047097 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260927 02:53:07.047700 8211 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43331
I20260927 02:53:07.047773 8211 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:07.047948 8211 heartbeater.cc:511] Master 127.0.98.62:43331 requested a full tablet report, sending...
I20260927 02:53:07.048269 7868 ts_manager.cc:194] Registered new tserver with Master: c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439)
I20260927 02:53:07.048646 7868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49747
I20260927 02:53:07.053419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 8080
I20260927 02:53:07.053536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:53:07.053573 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.048818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35549 (local address 127.0.98.62:43331)
0927 02:53:07.048858 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.048860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.048956 (+ 96us) server_negotiation.cc:407] Connection header received
0927 02:53:07.049014 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.049015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.049026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.049099 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.049357 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.049848 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.050501 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.050681 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.050737 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.051585 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:07.051586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:07.051587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:07.051593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:07.052633 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.052918 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.052919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.052919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.052973 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.053109 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.053109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.053109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.053256 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:07.053304 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.053432 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.053489 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:53:07.053607 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.048760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43331 (local address 127.0.98.2:35549)
0927 02:53:07.048884 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.048895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.048954 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.049099 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.049099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.049240 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:07.049295 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.049296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.049942 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.049943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.050426 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.050427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.050780 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.050781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.050860 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.051216 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:07.051219 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:07.051503 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:07.052701 (+ 1198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.052702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.052702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.052842 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.053031 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.053031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.053031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.053075 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.053304 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:07.053305 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:07.053346 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.053415 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.053506 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260927 02:53:07.053727 392 external_mini_cluster.cc:1428] Setting key 47642b38e2d459e08b48697a81361e11
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:53:07.077332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43331
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:07.156018 8220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:07.156213 8220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:07.156235 8220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:07.157307 8220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:07.157343 8220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:07.157362 8220 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:53:07.157382 8220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:07.158915 8220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:07.159155 8220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:07.159356 8220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:07.165171 8225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:07.165198 8228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:07.165745 8226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:07.166558 8220 server_base.cc:1034] running on GCE node
I20260927 02:53:07.166847 8220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:07.167120 8220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:07.168269 8220 hybrid_clock.cc:648] HybridClock initialized: now 1790477587168254 us; error 32 us; skew 500 ppm
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477587, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:07.170614 8220 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:53:07.171136 8220 webserver.cc:460] Webserver started at http://127.0.98.3:36229/ using document root <none> and password file <none>
I20260927 02:53:07.171337 8220 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:07.171386 8220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:07.171516 8220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:07.172545 8220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "e2e27d14e74a4d42a5dd1a974da19f9d"
format_stamp: "Formatted at 2026-09-27 02:53:07 on dist-test-slave-mfx9"
server_key: "a3598db52cba9754c3292ea4453b0966"
server_key_iv: "ae733fedf57392f879d4316d38dab27b"
server_key_version: "encryptionkey@0"
I20260927 02:53:07.172871 8220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e2e27d14e74a4d42a5dd1a974da19f9d"
format_stamp: "Formatted at 2026-09-27 02:53:07 on dist-test-slave-mfx9"
server_key: "a3598db52cba9754c3292ea4453b0966"
server_key_iv: "ae733fedf57392f879d4316d38dab27b"
server_key_version: "encryptionkey@0"
I20260927 02:53:07.174082 8220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:53:07.174789 8235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.174939 8220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:53:07.175019 8220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "e2e27d14e74a4d42a5dd1a974da19f9d"
format_stamp: "Formatted at 2026-09-27 02:53:07 on dist-test-slave-mfx9"
server_key: "a3598db52cba9754c3292ea4453b0966"
server_key_iv: "ae733fedf57392f879d4316d38dab27b"
server_key_version: "encryptionkey@0"
I20260927 02:53:07.175084 8220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:07.188278 8220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:07.190444 8220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:07.190608 8220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:07.191128 8220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:07.191171 8220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.191200 8220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:07.191217 8220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477587, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:07.197806 8220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35369
I20260927 02:53:07.197947 8350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35369 every 8 connection(s)
I20260927 02:53:07.198171 8220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:53:07.200796 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.191620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57891 (local address 127.0.98.62:43331)
0927 02:53:07.191665 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.191667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.191880 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:53:07.192446 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.192447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.192460 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.192533 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.193345 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.194091 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.195418 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.195670 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.195728 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.198228 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:07.198230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:07.198230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:07.198239 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:07.199634 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.200014 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.200014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.200015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.200060 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.200235 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.200235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.200236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.200334 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:07.200376 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.200674 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.200732 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:53:07.201035 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.191617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43331 (local address 127.0.98.3:57891)
0927 02:53:07.191755 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.191796 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.192322 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.192614 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.192617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.192969 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:07.193250 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.193256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.194179 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.194180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.195334 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.195337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.195746 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.195747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.195902 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.196503 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:07.196514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:07.198137 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:07.199707 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.199712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.199715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.199945 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.200101 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.200102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.200102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.200196 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.200418 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:07.200423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:07.200556 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.200651 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.200776 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260927 02:53:07.201458 8351 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43331
I20260927 02:53:07.201534 8351 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:07.201691 8351 heartbeater.cc:511] Master 127.0.98.62:43331 requested a full tablet report, sending...
I20260927 02:53:07.201874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 8220
I20260927 02:53:07.201985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:53:07.202175 392 external_mini_cluster.cc:1428] Setting key 8973a79f0690bd7ee903048e6f11234c
I20260927 02:53:07.202517 7868 ts_manager.cc:194] Registered new tserver with Master: e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369)
I20260927 02:53:07.203013 7868 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57891
I20260927 02:53:07.203078 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:53:07.207949 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.202606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42947 (local address 127.0.98.62:43331)
0927 02:53:07.202659 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.202661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.202698 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:53:07.202810 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.202811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.202820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.202862 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.203178 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.203852 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.204536 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.204782 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.204820 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.205798 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:07.205800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:07.205800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:07.205806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:07.207011 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.207287 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.207289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.207289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.207340 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.207492 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.207493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.207494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.207598 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:07.207651 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.207783 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.207852 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:53:07.207962 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.202473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43331 (local address 127.0.98.3:42947)
0927 02:53:07.202643 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.202653 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.202721 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.202865 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.202866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.203050 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:07.203125 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.203127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.203931 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.203932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.204433 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.204435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.204894 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.204895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.204956 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.205425 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:07.205431 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:07.205725 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:07.207078 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.207079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.207079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.207233 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.207390 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.207391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.207391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.207445 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.207647 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:07.207648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:07.207690 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.207757 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.207855 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:53:07.223639 7918 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:07.218425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.0.98.62:43331)
0927 02:53:07.218475 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.218477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.218487 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:53:07.218512 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.218512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.218520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.218568 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.219213 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.219709 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.220461 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.220667 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.220724 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.223562 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33868: BlockingRecv error: recv got EOF from 127.0.0.1:33868 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:07.230512 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.224272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.0.98.62:43331)
0927 02:53:07.224328 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.224330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.224397 (+ 67us) server_negotiation.cc:407] Connection header received
0927 02:53:07.224468 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.224468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.224478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.224518 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.224944 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.225460 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.226094 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.226307 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.226355 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.228334 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:07.228335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:07.228336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:07.228344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:07.229495 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.229794 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.229795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.229796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.229891 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.230054 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.230054 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.230055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.230161 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:07.230245 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.230345 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.230418 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:53:07.231840 7868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:53:07.233350 7868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:53:07.239383 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.235392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44439 (local address 127.0.0.1:46740)
0927 02:53:07.235650 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.235660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.235710 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.236013 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.236014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.236016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.236070 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.236071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.237027 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.237028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.238272 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.238274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.239085 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.239086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.239163 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.239176 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.239247 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.239297 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":24,"thread_start_us":86,"threads_started":1}
I20260927 02:53:07.239398 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.235491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.0.98.2:44439)
0927 02:53:07.235823 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.235828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.235837 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:07.235869 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.235871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.235883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.235943 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.236120 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.236957 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.238358 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.239011 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.239077 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.239155 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.239262 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.239297 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":77,"threads_started":1}
I20260927 02:53:07.239590 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.235643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34077 (local address 127.0.0.1:57760)
0927 02:53:07.235885 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.235895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.235941 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.236388 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.236388 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.236390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.236425 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.236426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.237538 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.237539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.238454 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.238456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.239377 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.239378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.239428 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.239431 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.239485 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.239520 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":64,"threads_started":1}
I20260927 02:53:07.239822 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.235855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.98.1:34077)
0927 02:53:07.236196 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.236200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.236208 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:07.236241 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.236243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.236257 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.236327 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.236496 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.237472 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.238529 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.239230 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.239285 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.239351 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.239693 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.239735 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":76,"threads_started":1}
I20260927 02:53:07.240375 8005 tablet_service.cc:1467] Processing CreateTablet for tablet 9f316c4ac54a4633abd2f1758a35ab14 (DEFAULT_TABLE table=test-table [id=162db6ad309949eb94a23a65b0a68fd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.240362 8145 tablet_service.cc:1467] Processing CreateTablet for tablet 9f316c4ac54a4633abd2f1758a35ab14 (DEFAULT_TABLE table=test-table [id=162db6ad309949eb94a23a65b0a68fd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.240733 8145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f316c4ac54a4633abd2f1758a35ab14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.240733 8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f316c4ac54a4633abd2f1758a35ab14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.244513 8379 tablet_bootstrap.cc:492] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Bootstrap starting.
I20260927 02:53:07.244599 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.235384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35369 (local address 127.0.0.1:45756)
0927 02:53:07.235583 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.235624 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.235683 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.235947 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.235949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.235952 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.236013 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.236014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.237276 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.237276 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.238326 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.238328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.239296 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.239297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.239365 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.239370 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.239413 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.244529 (+ 5116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":47,"threads_started":1}
I20260927 02:53:07.244832 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.235559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.0.98.3:35369)
0927 02:53:07.235736 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.235741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.235752 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:53:07.235790 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.235792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.235807 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.235876 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.236087 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.237187 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.238390 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.239115 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.239182 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.239263 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.244706 (+ 5443us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.244759 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260927 02:53:07.245260 8379 tablet_bootstrap.cc:654] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.245581 8379 log.cc:824] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:07.245466 8285 tablet_service.cc:1467] Processing CreateTablet for tablet 9f316c4ac54a4633abd2f1758a35ab14 (DEFAULT_TABLE table=test-table [id=162db6ad309949eb94a23a65b0a68fd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.245880 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f316c4ac54a4633abd2f1758a35ab14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.245927 8380 tablet_bootstrap.cc:492] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Bootstrap starting.
I20260927 02:53:07.246230 8379 tablet_bootstrap.cc:492] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: No bootstrap required, opened a new log
I20260927 02:53:07.246302 8379 ts_tablet_manager.cc:1397] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:53:07.246541 8380 tablet_bootstrap.cc:654] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.247007 8380 log.cc:824] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:07.247782 8380 tablet_bootstrap.cc:492] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: No bootstrap required, opened a new log
I20260927 02:53:07.247850 8380 ts_tablet_manager.cc:1397] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:53:07.248092 8379 raft_consensus.cc:348] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.248216 8379 raft_consensus.cc:374] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.248246 8379 raft_consensus.cc:729] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e02f9699042f6b406d52d0fc3e664, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.248378 8379 consensus_queue.cc:260] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.248621 8379 ts_tablet_manager.cc:1428] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:53:07.248751 8211 heartbeater.cc:503] Master 127.0.98.62:43331 was elected leader, sending a full tablet report...
I20260927 02:53:07.249604 8384 tablet_bootstrap.cc:492] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: Bootstrap starting.
I20260927 02:53:07.249743 8380 raft_consensus.cc:348] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.249903 8380 raft_consensus.cc:374] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.249943 8380 raft_consensus.cc:729] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34282bd6edb14a04a35651b4b019dafd, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.250051 8380 consensus_queue.cc:260] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.250180 8384 tablet_bootstrap.cc:654] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.250306 8380 ts_tablet_manager.cc:1428] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:53:07.250314 8071 heartbeater.cc:503] Master 127.0.98.62:43331 was elected leader, sending a full tablet report...
I20260927 02:53:07.250483 8384 log.cc:824] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:07.251415 8384 tablet_bootstrap.cc:492] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: No bootstrap required, opened a new log
I20260927 02:53:07.251472 8384 ts_tablet_manager.cc:1397] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:53:07.252700 8384 raft_consensus.cc:348] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.252813 8384 raft_consensus.cc:374] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.252833 8384 raft_consensus.cc:729] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e27d14e74a4d42a5dd1a974da19f9d, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.252911 8384 consensus_queue.cc:260] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.253111 8384 ts_tablet_manager.cc:1428] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:53:07.253162 8351 heartbeater.cc:503] Master 127.0.98.62:43331 was elected leader, sending a full tablet report...
W20260927 02:53:07.294925 8212 tablet.cc:2369] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:53:07.385783 8383 raft_consensus.cc:482] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:53:07.385947 8383 raft_consensus.cc:504] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.386291 8383 leader_election.cc:288] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369), 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077)
I20260927 02:53:07.389379 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.386689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50415 (local address 127.0.98.1:34077)
0927 02:53:07.386727 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.386729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.386806 (+ 77us) server_negotiation.cc:407] Connection header received
0927 02:53:07.386908 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.386909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.386921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.386949 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.387131 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.387765 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.388715 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.389076 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.389139 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.389148 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.389263 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.389298 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:53:07.389379 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.386535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38293 (local address 127.0.98.3:35369)
0927 02:53:07.386584 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.386586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.386597 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:53:07.386718 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.386720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.386732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.386857 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.387073 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.387723 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.388702 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.389058 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.389110 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.389121 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.389279 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.389313 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:53:07.389386 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.386459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35369 (local address 127.0.98.2:38293)
0927 02:53:07.386548 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.386560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.386606 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.386891 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.386892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.386894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.386980 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.386982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.387796 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.387797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.388619 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.388622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.389130 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.389131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.389217 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.389219 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.389267 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.389302 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 02:53:07.389400 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.386460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34077 (local address 127.0.98.2:50415)
0927 02:53:07.386721 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.386732 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.386830 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.387009 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.387009 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.387010 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.387073 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.387074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.387837 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.387838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.388630 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.388633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.389154 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.389155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.389207 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.389210 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.389248 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.389285 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":13,"thread_start_us":113,"threads_started":1}
I20260927 02:53:07.389799 8305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f316c4ac54a4633abd2f1758a35ab14" candidate_uuid: "c84e02f9699042f6b406d52d0fc3e664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" is_pre_election: true
I20260927 02:53:07.389882 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f316c4ac54a4633abd2f1758a35ab14" candidate_uuid: "c84e02f9699042f6b406d52d0fc3e664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34282bd6edb14a04a35651b4b019dafd" is_pre_election: true
I20260927 02:53:07.389942 8305 raft_consensus.cc:2454] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84e02f9699042f6b406d52d0fc3e664 in term 0.
I20260927 02:53:07.390027 8025 raft_consensus.cc:2454] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84e02f9699042f6b406d52d0fc3e664 in term 0.
I20260927 02:53:07.390153 8096 leader_election.cc:302] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c84e02f9699042f6b406d52d0fc3e664, e2e27d14e74a4d42a5dd1a974da19f9d; no voters:
I20260927 02:53:07.390280 8383 raft_consensus.cc:2781] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:53:07.390329 8383 raft_consensus.cc:482] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:53:07.390352 8383 raft_consensus.cc:3037] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.391448 8383 raft_consensus.cc:504] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.391577 8383 leader_election.cc:288] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [CANDIDATE]: Term 1 election: Requested vote from peers e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369), 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077)
I20260927 02:53:07.391758 8305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f316c4ac54a4633abd2f1758a35ab14" candidate_uuid: "c84e02f9699042f6b406d52d0fc3e664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d"
I20260927 02:53:07.391779 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f316c4ac54a4633abd2f1758a35ab14" candidate_uuid: "c84e02f9699042f6b406d52d0fc3e664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34282bd6edb14a04a35651b4b019dafd"
I20260927 02:53:07.391826 8305 raft_consensus.cc:3037] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.391846 8025 raft_consensus.cc:3037] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.392478 8305 raft_consensus.cc:2454] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84e02f9699042f6b406d52d0fc3e664 in term 1.
I20260927 02:53:07.392593 8025 raft_consensus.cc:2454] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84e02f9699042f6b406d52d0fc3e664 in term 1.
I20260927 02:53:07.392668 8096 leader_election.cc:302] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c84e02f9699042f6b406d52d0fc3e664, e2e27d14e74a4d42a5dd1a974da19f9d; no voters:
I20260927 02:53:07.392757 8383 raft_consensus.cc:2781] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:07.392920 8383 raft_consensus.cc:686] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 1 LEADER]: Becoming Leader. State: Replica: c84e02f9699042f6b406d52d0fc3e664, State: Running, Role: LEADER
I20260927 02:53:07.393003 8383 consensus_queue.cc:237] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260927 02:53:07.393769 7867 catalog_manager.cc:5495] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 reported cstate change: term changed from 0 to 1, leader changed from <none> to c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c84e02f9699042f6b406d52d0fc3e664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } health_report { overall_health: UNKNOWN } } }
W20260927 02:53:07.400818 8072 tablet.cc:2369] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:53:07.419018 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.416753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.0.98.2:44439)
0927 02:53:07.416810 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.416812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.416819 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:53:07.416891 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.416892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.416912 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.416974 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:07.417103 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.417565 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.418293 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.418477 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.418532 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.418683 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:07.418820 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:07.418895 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.418925 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.418944 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:53:07.420688 8305 raft_consensus.cc:1261] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Refusing update from remote peer c84e02f9699042f6b406d52d0fc3e664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:07.420688 8025 raft_consensus.cc:1261] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Refusing update from remote peer c84e02f9699042f6b406d52d0fc3e664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:07.420993 8383 consensus_queue.cc:1035] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:07.421113 8389 consensus_queue.cc:1035] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:07.423210 8392 mvcc.cc:205] Tried to move back new op lower bound from 7333796198072487936 to 7333796197962272768. Current Snapshot: MvccSnapshot[applied={T|T < 7333796198072487936}]
I20260927 02:53:07.423911 8394 mvcc.cc:205] Tried to move back new op lower bound from 7333796198072487936 to 7333796197962272768. Current Snapshot: MvccSnapshot[applied={T|T < 7333796198072487936}]
I20260927 02:53:07.425113 7867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:07.425208 7867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:07.425827 7867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: Sending DeleteTablet for 3 replicas of tablet 9f316c4ac54a4633abd2f1758a35ab14
I20260927 02:53:07.426108 8285 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f316c4ac54a4633abd2f1758a35ab14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260927 02:53:07.426141 8005 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f316c4ac54a4633abd2f1758a35ab14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57760
I20260927 02:53:07.426206 8145 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f316c4ac54a4633abd2f1758a35ab14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46740
I20260927 02:53:07.426388 8403 tablet_replica.cc:330] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: stopping tablet replica
I20260927 02:53:07.426436 8402 tablet_replica.cc:330] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: stopping tablet replica
I20260927 02:53:07.426450 7867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:53:07.426537 8402 raft_consensus.cc:2229] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 02:53:07.426577 7867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:53:07.426695 8403 raft_consensus.cc:2229] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:07.426898 8402 raft_consensus.cc:2258] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.426872 8403 raft_consensus.cc:2258] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.427270 8404 tablet_replica.cc:330] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: stopping tablet replica
I20260927 02:53:07.427353 8404 raft_consensus.cc:2229] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:53:07.427433 8404 raft_consensus.cc:2258] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.427502 8402 ts_tablet_manager.cc:1905] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.427809 8404 ts_tablet_manager.cc:1905] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.428093 8005 tablet_service.cc:1467] Processing CreateTablet for tablet cfd8be86456a47f3b84356ed4f470dee (DEFAULT_TABLE table=test-table [id=443783ac3f6442c7a11fc5db61237745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.428186 8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd8be86456a47f3b84356ed4f470dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.429070 8402 ts_tablet_manager.cc:1918] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:07.429127 8402 log.cc:1196] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/9f316c4ac54a4633abd2f1758a35ab14
I20260927 02:53:07.429306 8404 ts_tablet_manager.cc:1918] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:07.429355 8402 ts_tablet_manager.cc:1939] T 9f316c4ac54a4633abd2f1758a35ab14 P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting consensus metadata
I20260927 02:53:07.429364 8404 log.cc:1196] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/9f316c4ac54a4633abd2f1758a35ab14
I20260927 02:53:07.429435 8380 tablet_bootstrap.cc:492] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: Bootstrap starting.
I20260927 02:53:07.429610 8404 ts_tablet_manager.cc:1939] T 9f316c4ac54a4633abd2f1758a35ab14 P c84e02f9699042f6b406d52d0fc3e664: Deleting consensus metadata
I20260927 02:53:07.429770 8380 tablet_bootstrap.cc:654] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.430186 7853 catalog_manager.cc:4841] TS c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439): tablet 9f316c4ac54a4633abd2f1758a35ab14 (table test-table [id=162db6ad309949eb94a23a65b0a68fd8]) successfully deleted
I20260927 02:53:07.430415 8145 tablet_service.cc:1467] Processing CreateTablet for tablet cfd8be86456a47f3b84356ed4f470dee (DEFAULT_TABLE table=test-table [id=443783ac3f6442c7a11fc5db61237745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.430509 8145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd8be86456a47f3b84356ed4f470dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.430914 7852 catalog_manager.cc:4841] TS e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369): tablet 9f316c4ac54a4633abd2f1758a35ab14 (table test-table [id=162db6ad309949eb94a23a65b0a68fd8]) successfully deleted
I20260927 02:53:07.431116 8285 tablet_service.cc:1467] Processing CreateTablet for tablet cfd8be86456a47f3b84356ed4f470dee (DEFAULT_TABLE table=test-table [id=443783ac3f6442c7a11fc5db61237745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.431212 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd8be86456a47f3b84356ed4f470dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.431488 8379 tablet_bootstrap.cc:492] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: Bootstrap starting.
I20260927 02:53:07.431833 8379 tablet_bootstrap.cc:654] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.432282 8384 tablet_bootstrap.cc:492] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: Bootstrap starting.
I20260927 02:53:07.432329 8379 tablet_bootstrap.cc:492] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: No bootstrap required, opened a new log
I20260927 02:53:07.432371 8379 ts_tablet_manager.cc:1397] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:07.432426 8380 tablet_bootstrap.cc:492] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: No bootstrap required, opened a new log
I20260927 02:53:07.432475 8380 ts_tablet_manager.cc:1397] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:53:07.432487 8379 raft_consensus.cc:348] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.432530 8379 raft_consensus.cc:374] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.432541 8379 raft_consensus.cc:729] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e02f9699042f6b406d52d0fc3e664, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.432575 8379 consensus_queue.cc:260] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.432641 8384 tablet_bootstrap.cc:654] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.432605 8380 raft_consensus.cc:348] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.432648 8379 ts_tablet_manager.cc:1428] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.432665 8380 raft_consensus.cc:374] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.432685 8380 raft_consensus.cc:729] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34282bd6edb14a04a35651b4b019dafd, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.432727 8380 consensus_queue.cc:260] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.432813 8380 ts_tablet_manager.cc:1428] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.433259 8403 ts_tablet_manager.cc:1905] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.434490 8403 ts_tablet_manager.cc:1918] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:07.434551 8403 log.cc:1196] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/9f316c4ac54a4633abd2f1758a35ab14
I20260927 02:53:07.435020 8403 ts_tablet_manager.cc:1939] T 9f316c4ac54a4633abd2f1758a35ab14 P 34282bd6edb14a04a35651b4b019dafd: Deleting consensus metadata
I20260927 02:53:07.435321 7853 catalog_manager.cc:4841] TS 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077): tablet 9f316c4ac54a4633abd2f1758a35ab14 (table test-table [id=162db6ad309949eb94a23a65b0a68fd8]) successfully deleted
I20260927 02:53:07.439145 8384 tablet_bootstrap.cc:492] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: No bootstrap required, opened a new log
I20260927 02:53:07.439193 8384 ts_tablet_manager.cc:1397] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 02:53:07.439344 8384 raft_consensus.cc:348] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.439404 8384 raft_consensus.cc:374] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.439415 8384 raft_consensus.cc:729] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e27d14e74a4d42a5dd1a974da19f9d, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.439455 8384 consensus_queue.cc:260] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.439546 8384 ts_tablet_manager.cc:1428] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260927 02:53:07.448670 8352 tablet.cc:2369] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:53:07.508018 8387 raft_consensus.cc:482] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:53:07.508123 8387 raft_consensus.cc:504] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.508425 8387 leader_election.cc:288] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077), c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439)
I20260927 02:53:07.511648 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.508877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44265 (local address 127.0.98.2:44439)
0927 02:53:07.508919 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.508921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.508928 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:53:07.508957 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.508958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.508966 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.509007 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.509235 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.509946 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.510917 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.511304 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.511370 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.511380 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.511542 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.511576 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:53:07.511691 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.508737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44439 (local address 127.0.98.3:44265)
0927 02:53:07.508867 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.508876 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.508915 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.509055 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.509057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.509059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.509154 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.509157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.510002 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.510003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.510842 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.510844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.511397 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.511398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.511468 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.511473 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.511544 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.511581 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
I20260927 02:53:07.512033 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.508639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34077 (local address 127.0.98.3:45893)
0927 02:53:07.508693 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.508706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.508752 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.508987 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.508988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.508989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.509059 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.509060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.509985 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.509986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.511164 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.511167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.511796 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.511797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.511886 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.511889 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.511932 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.511970 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":35}
I20260927 02:53:07.512094 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.508705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45893 (local address 127.0.98.1:34077)
0927 02:53:07.508770 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.508772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.508784 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:53:07.508866 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.508868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.508884 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.508974 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.509128 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.509886 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.511258 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.511707 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.511770 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.511780 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.512001 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.512036 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:53:07.512238 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd8be86456a47f3b84356ed4f470dee" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34282bd6edb14a04a35651b4b019dafd" is_pre_election: true
I20260927 02:53:07.512223 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd8be86456a47f3b84356ed4f470dee" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e02f9699042f6b406d52d0fc3e664" is_pre_election: true
I20260927 02:53:07.512315 8025 raft_consensus.cc:2454] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 0.
I20260927 02:53:07.512327 8165 raft_consensus.cc:2454] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 0.
I20260927 02:53:07.512490 8237 leader_election.cc:302] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c84e02f9699042f6b406d52d0fc3e664, e2e27d14e74a4d42a5dd1a974da19f9d; no voters:
I20260927 02:53:07.512612 8387 raft_consensus.cc:2781] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:53:07.512652 8387 raft_consensus.cc:482] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:53:07.512671 8387 raft_consensus.cc:3037] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.513317 8387 raft_consensus.cc:504] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.513419 8387 leader_election.cc:288] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 election: Requested vote from peers 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077), c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439)
I20260927 02:53:07.513569 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd8be86456a47f3b84356ed4f470dee" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34282bd6edb14a04a35651b4b019dafd"
I20260927 02:53:07.513638 8025 raft_consensus.cc:3037] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.513648 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd8be86456a47f3b84356ed4f470dee" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e02f9699042f6b406d52d0fc3e664"
I20260927 02:53:07.513710 8165 raft_consensus.cc:3037] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.514339 8025 raft_consensus.cc:2454] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 1.
I20260927 02:53:07.514411 8165 raft_consensus.cc:2454] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 1.
I20260927 02:53:07.514511 8237 leader_election.cc:302] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34282bd6edb14a04a35651b4b019dafd, e2e27d14e74a4d42a5dd1a974da19f9d; no voters:
I20260927 02:53:07.514587 8387 raft_consensus.cc:2781] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:07.514732 8387 raft_consensus.cc:686] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 LEADER]: Becoming Leader. State: Replica: e2e27d14e74a4d42a5dd1a974da19f9d, State: Running, Role: LEADER
I20260927 02:53:07.514837 8387 consensus_queue.cc:237] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.515548 7867 catalog_manager.cc:5495] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } health_report { overall_health: HEALTHY } } }
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477586, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:07.557678 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.551485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43331 (local address 127.0.0.1:33896)
0927 02:53:07.551639 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.551648 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.551710 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.551857 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.551858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.552149 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:07.552196 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.552198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.552840 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.552841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.553367 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.553368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.553658 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.553659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.553703 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.554198 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:07.554202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:07.555488 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:07.556786 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.556830 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.556832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.557019 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.557175 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:07.557176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:07.557176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:07.557223 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:07.557381 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:07.557384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:07.557421 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.557464 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.557546 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260927 02:53:07.557678 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.551530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.0.98.62:43331)
0927 02:53:07.551586 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.551588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.551704 (+ 116us) server_negotiation.cc:407] Connection header received
0927 02:53:07.551751 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.551752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.551761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.551819 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.552250 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.552726 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.553429 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.553604 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.553644 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.555555 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:07.555556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:07.555556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:07.555562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:07.556718 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.557081 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.557082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.557082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.557128 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.557262 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.557262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.557262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.557348 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:07.557389 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.557491 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.557585 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":34}
I20260927 02:53:07.559657 7867 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33896:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:53:07.559831 7867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:53:07.561136 8285 tablet_service.cc:1467] Processing CreateTablet for tablet da7670a4b4b54a678baf4cd5b259f236 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62dbd1bd54c64bfe9e0c75be5681fff4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:53:07.561169 8005 tablet_service.cc:1467] Processing CreateTablet for tablet da7670a4b4b54a678baf4cd5b259f236 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62dbd1bd54c64bfe9e0c75be5681fff4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:53:07.561290 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7670a4b4b54a678baf4cd5b259f236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.561322 8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7670a4b4b54a678baf4cd5b259f236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.562330 8384 tablet_bootstrap.cc:492] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: Bootstrap starting.
I20260927 02:53:07.562477 8145 tablet_service.cc:1467] Processing CreateTablet for tablet da7670a4b4b54a678baf4cd5b259f236 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62dbd1bd54c64bfe9e0c75be5681fff4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:53:07.562625 8145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7670a4b4b54a678baf4cd5b259f236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.562716 8384 tablet_bootstrap.cc:654] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.563346 8384 tablet_bootstrap.cc:492] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: No bootstrap required, opened a new log
I20260927 02:53:07.563395 8384 ts_tablet_manager.cc:1397] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:53:07.563527 8384 raft_consensus.cc:348] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.563598 8384 raft_consensus.cc:374] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.563616 8384 raft_consensus.cc:729] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e27d14e74a4d42a5dd1a974da19f9d, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.563655 8384 consensus_queue.cc:260] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.563742 8384 ts_tablet_manager.cc:1428] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.563782 8387 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.563843 8387 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:53:07.563874 8380 tablet_bootstrap.cc:492] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: Bootstrap starting.
I20260927 02:53:07.564199 8380 tablet_bootstrap.cc:654] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.564509 8379 tablet_bootstrap.cc:492] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: Bootstrap starting.
I20260927 02:53:07.564754 8380 tablet_bootstrap.cc:492] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: No bootstrap required, opened a new log
I20260927 02:53:07.564787 8379 tablet_bootstrap.cc:654] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.564798 8380 ts_tablet_manager.cc:1397] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:07.564920 8380 raft_consensus.cc:348] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.564972 8380 raft_consensus.cc:374] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.564990 8380 raft_consensus.cc:729] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34282bd6edb14a04a35651b4b019dafd, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.565028 8380 consensus_queue.cc:260] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.565109 8380 ts_tablet_manager.cc:1428] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.565265 8379 tablet_bootstrap.cc:492] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: No bootstrap required, opened a new log
I20260927 02:53:07.565223 8385 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.565294 8385 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:53:07.565302 8379 ts_tablet_manager.cc:1397] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:07.565392 8379 raft_consensus.cc:348] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.565447 8379 raft_consensus.cc:374] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.565465 8379 raft_consensus.cc:729] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e02f9699042f6b406d52d0fc3e664, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.565501 8379 consensus_queue.cc:260] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.565565 8379 ts_tablet_manager.cc:1428] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.565599 8389 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.565693 8389 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:53:07.697489 8387 raft_consensus.cc:482] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:53:07.697579 8387 raft_consensus.cc:504] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.697718 8387 leader_election.cc:288] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077), c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439)
I20260927 02:53:07.697994 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7670a4b4b54a678baf4cd5b259f236" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34282bd6edb14a04a35651b4b019dafd" is_pre_election: true
I20260927 02:53:07.698002 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7670a4b4b54a678baf4cd5b259f236" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e02f9699042f6b406d52d0fc3e664" is_pre_election: true
I20260927 02:53:07.698077 8025 raft_consensus.cc:2454] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 0.
I20260927 02:53:07.698096 8165 raft_consensus.cc:2454] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 0.
I20260927 02:53:07.698220 8237 leader_election.cc:302] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34282bd6edb14a04a35651b4b019dafd, e2e27d14e74a4d42a5dd1a974da19f9d; no voters:
I20260927 02:53:07.698320 8387 raft_consensus.cc:2781] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:53:07.698362 8387 raft_consensus.cc:482] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:53:07.698376 8387 raft_consensus.cc:3037] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.699067 8387 raft_consensus.cc:504] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.699167 8387 leader_election.cc:288] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 election: Requested vote from peers 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077), c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439)
I20260927 02:53:07.699361 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7670a4b4b54a678baf4cd5b259f236" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34282bd6edb14a04a35651b4b019dafd"
I20260927 02:53:07.699383 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7670a4b4b54a678baf4cd5b259f236" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e02f9699042f6b406d52d0fc3e664"
I20260927 02:53:07.699445 8165 raft_consensus.cc:3037] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.699442 8025 raft_consensus.cc:3037] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.700137 8165 raft_consensus.cc:2454] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 1.
I20260927 02:53:07.700138 8025 raft_consensus.cc:2454] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 1.
I20260927 02:53:07.700323 8237 leader_election.cc:302] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34282bd6edb14a04a35651b4b019dafd, e2e27d14e74a4d42a5dd1a974da19f9d; no voters:
I20260927 02:53:07.700423 8387 raft_consensus.cc:2781] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:07.700471 8387 raft_consensus.cc:686] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 LEADER]: Becoming Leader. State: Replica: e2e27d14e74a4d42a5dd1a974da19f9d, State: Running, Role: LEADER
I20260927 02:53:07.700526 8387 consensus_queue.cc:237] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.701013 8409 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: TxnStatusTablet state changed. Reason: New leader e2e27d14e74a4d42a5dd1a974da19f9d. Latest consensus state: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.701098 8409 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:53:07.701061 7867 catalog_manager.cc:5495] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } health_report { overall_health: HEALTHY } } }
I20260927 02:53:07.701264 8421 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:53:07.701397 8421 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:53:07.709856 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.707397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35369 (local address 127.0.0.1:45766)
0927 02:53:07.707509 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.707519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.707572 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.707725 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.707726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.707727 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:53:07.707773 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.707775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.708449 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.708450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.709028 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.709030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.709344 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.709345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.709409 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.709425 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:53:07.709726 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:53:07.709731 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.709757 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.709797 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 02:53:07.709872 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.707444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.0.98.3:35369)
0927 02:53:07.707495 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.707497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.707553 (+ 56us) server_negotiation.cc:407] Connection header received
0927 02:53:07.707644 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.707644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.707661 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.707711 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:07.707831 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.708379 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.709104 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.709288 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.709351 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.709530 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:07.709661 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:07.709747 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.709772 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.709796 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:53:07.711012 8025 raft_consensus.cc:1261] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Refusing update from remote peer e2e27d14e74a4d42a5dd1a974da19f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:07.711097 8165 raft_consensus.cc:1261] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Refusing update from remote peer e2e27d14e74a4d42a5dd1a974da19f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:07.711409 8387 consensus_queue.cc:1035] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:07.711541 8409 consensus_queue.cc:1035] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:07.712132 8385 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: TxnStatusTablet state changed. Reason: New leader e2e27d14e74a4d42a5dd1a974da19f9d. Latest consensus state: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.712220 8385 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:53:07.712179 8389 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: TxnStatusTablet state changed. Reason: New leader e2e27d14e74a4d42a5dd1a974da19f9d. Latest consensus state: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.712260 8389 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:53:07.712870 8409 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.712945 8409 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:53:07.713107 8387 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.713162 8387 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P e2e27d14e74a4d42a5dd1a974da19f9d: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:53:07.713455 8389 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.713541 8389 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P c84e02f9699042f6b406d52d0fc3e664: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:53:07.714146 8385 tablet_replica.cc:439] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } } }
I20260927 02:53:07.714231 8385 tablet_replica.cc:442] T da7670a4b4b54a678baf4cd5b259f236 P 34282bd6edb14a04a35651b4b019dafd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:53:07.716425 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.714305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.0.98.3:35369)
0927 02:53:07.714344 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.714346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.714352 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:53:07.714370 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.714371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.714380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.714423 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:07.714589 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.715149 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.715844 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.716044 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.716096 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.716224 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:07.716257 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:07.716299 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.716346 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.716363 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:53:07.721004 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.718533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35147 (local address 127.0.98.3:35369)
0927 02:53:07.718568 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.718569 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.718676 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:53:07.718735 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.718735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.718742 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.718816 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:07.718943 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.719355 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.720425 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.720601 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.720644 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.720807 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:07.720830 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:07.720859 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.720909 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.720926 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260927 02:53:07.721101 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.718470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35369 (local address 127.0.98.3:35147)
0927 02:53:07.718614 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.718624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.718682 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.718818 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.718819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.718820 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:53:07.718889 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.718890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.719412 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.719412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.720357 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.720360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.720659 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.720660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.720724 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.720740 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:53:07.720862 (+ 122us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:53:07.720864 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.720907 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.720950 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260927 02:53:07.723325 8165 raft_consensus.cc:1261] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Refusing update from remote peer e2e27d14e74a4d42a5dd1a974da19f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:07.723369 8025 raft_consensus.cc:1261] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Refusing update from remote peer e2e27d14e74a4d42a5dd1a974da19f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:07.723558 8429 consensus_queue.cc:1035] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:07.723656 8387 consensus_queue.cc:1035] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:07.725186 8393 mvcc.cc:205] Tried to move back new op lower bound from 7333796199313731584 to 7333796198461300736. Current Snapshot: MvccSnapshot[applied={T|T < 7333796199313731584}]
I20260927 02:53:07.734814 7868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:07.734908 7868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:07.735438 7868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: Sending DeleteTablet for 3 replicas of tablet cfd8be86456a47f3b84356ed4f470dee
I20260927 02:53:07.735651 8005 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd8be86456a47f3b84356ed4f470dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57760
I20260927 02:53:07.735674 8145 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd8be86456a47f3b84356ed4f470dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46740
I20260927 02:53:07.735737 8403 tablet_replica.cc:330] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: stopping tablet replica
I20260927 02:53:07.735777 8403 raft_consensus.cc:2229] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:07.735810 8403 raft_consensus.cc:2258] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.735872 8404 tablet_replica.cc:330] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: stopping tablet replica
I20260927 02:53:07.735924 8404 raft_consensus.cc:2229] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:07.735955 8404 raft_consensus.cc:2258] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.736449 8403 ts_tablet_manager.cc:1905] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.738049 8284 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd8be86456a47f3b84356ed4f470dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260927 02:53:07.738139 8402 tablet_replica.cc:330] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: stopping tablet replica
I20260927 02:53:07.738178 8402 raft_consensus.cc:2229] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:53:07.738255 8402 raft_consensus.cc:2258] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.738582 8402 ts_tablet_manager.cc:1905] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.738950 8403 ts_tablet_manager.cc:1918] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:53:07.739001 8403 log.cc:1196] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/cfd8be86456a47f3b84356ed4f470dee
I20260927 02:53:07.739187 8403 ts_tablet_manager.cc:1939] T cfd8be86456a47f3b84356ed4f470dee P 34282bd6edb14a04a35651b4b019dafd: Deleting consensus metadata
I20260927 02:53:07.740051 8402 ts_tablet_manager.cc:1918] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:53:07.740096 8402 log.cc:1196] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/cfd8be86456a47f3b84356ed4f470dee
I20260927 02:53:07.740276 8402 ts_tablet_manager.cc:1939] T cfd8be86456a47f3b84356ed4f470dee P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting consensus metadata
I20260927 02:53:07.740597 7853 catalog_manager.cc:4841] TS 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077): tablet cfd8be86456a47f3b84356ed4f470dee (table test-table [id=443783ac3f6442c7a11fc5db61237745]) successfully deleted
I20260927 02:53:07.740818 8404 ts_tablet_manager.cc:1905] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.742249 8404 ts_tablet_manager.cc:1918] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:53:07.742305 8404 log.cc:1196] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/cfd8be86456a47f3b84356ed4f470dee
I20260927 02:53:07.742488 8404 ts_tablet_manager.cc:1939] T cfd8be86456a47f3b84356ed4f470dee P c84e02f9699042f6b406d52d0fc3e664: Deleting consensus metadata
I20260927 02:53:07.742781 7852 catalog_manager.cc:4841] TS e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369): tablet cfd8be86456a47f3b84356ed4f470dee (table test-table [id=443783ac3f6442c7a11fc5db61237745]) successfully deleted
I20260927 02:53:07.743077 7853 catalog_manager.cc:4841] TS c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439): tablet cfd8be86456a47f3b84356ed4f470dee (table test-table [id=443783ac3f6442c7a11fc5db61237745]) successfully deleted
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:53:07.756225 7918 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:07.750751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.0.98.62:43331)
0927 02:53:07.750828 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.750830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.750934 (+ 104us) server_negotiation.cc:407] Connection header received
0927 02:53:07.750991 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.750992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.751005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.751061 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.751572 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.752344 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.753030 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.753337 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.753402 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.756159 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33906: BlockingRecv error: recv got EOF from 127.0.0.1:33906 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 02:53:07 dist-test-slave-mfx9 krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:53:07.763371 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.756940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.0.98.62:43331)
0927 02:53:07.756981 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.756983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.757017 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:53:07.757085 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.757086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.757095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.757138 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:07.757552 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.758007 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.758620 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.758857 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.758905 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.761069 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:07.761071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:07.761071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:07.761076 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:07.762325 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.762694 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.762695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.762695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.762745 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:07.762982 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:07.762982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:07.762983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:07.763081 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:07.763149 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.763259 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.763311 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:53:07.764719 7868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:53:07.764860 7868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:53:07.766119 8145 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 (DEFAULT_TABLE table=test-table [id=cadbfdd4c7744727913adffe013da97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.766119 8284 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 (DEFAULT_TABLE table=test-table [id=cadbfdd4c7744727913adffe013da97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.766125 8005 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 (DEFAULT_TABLE table=test-table [id=cadbfdd4c7744727913adffe013da97f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.766237 8284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.766237 8145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.766244 8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.767544 8379 tablet_bootstrap.cc:492] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: Bootstrap starting.
I20260927 02:53:07.767570 8384 tablet_bootstrap.cc:492] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: Bootstrap starting.
I20260927 02:53:07.767566 8380 tablet_bootstrap.cc:492] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: Bootstrap starting.
I20260927 02:53:07.767879 8379 tablet_bootstrap.cc:654] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.767886 8384 tablet_bootstrap.cc:654] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.767926 8380 tablet_bootstrap.cc:654] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.768492 8384 tablet_bootstrap.cc:492] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: No bootstrap required, opened a new log
I20260927 02:53:07.768543 8384 ts_tablet_manager.cc:1397] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:53:07.768702 8384 raft_consensus.cc:348] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.768771 8384 raft_consensus.cc:374] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.768792 8384 raft_consensus.cc:729] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e27d14e74a4d42a5dd1a974da19f9d, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.768800 8379 tablet_bootstrap.cc:492] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: No bootstrap required, opened a new log
I20260927 02:53:07.768834 8379 ts_tablet_manager.cc:1397] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:07.768828 8384 consensus_queue.cc:260] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.768920 8384 ts_tablet_manager.cc:1428] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.768940 8380 tablet_bootstrap.cc:492] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: No bootstrap required, opened a new log
I20260927 02:53:07.768936 8379 raft_consensus.cc:348] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.768980 8380 ts_tablet_manager.cc:1397] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:07.768989 8379 raft_consensus.cc:374] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.769008 8379 raft_consensus.cc:729] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e02f9699042f6b406d52d0fc3e664, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.769042 8379 consensus_queue.cc:260] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.769124 8379 ts_tablet_manager.cc:1428] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.769109 8380 raft_consensus.cc:348] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.769169 8380 raft_consensus.cc:374] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.769186 8380 raft_consensus.cc:729] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34282bd6edb14a04a35651b4b019dafd, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.769229 8380 consensus_queue.cc:260] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.769326 8380 ts_tablet_manager.cc:1428] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.814494 8385 raft_consensus.cc:482] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:53:07.814604 8385 raft_consensus.cc:504] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.814991 8385 leader_election.cc:288] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439), e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369)
I20260927 02:53:07.818428 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.815135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44439 (local address 127.0.98.1:49175)
0927 02:53:07.815271 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.815283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.815369 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.815570 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.815571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.815574 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.815659 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.815661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.816520 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.816521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.817319 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.817321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.818123 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.818125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.818194 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.818200 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.818291 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.818335 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260927 02:53:07.818430 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.815254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49175 (local address 127.0.98.2:44439)
0927 02:53:07.815297 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.815299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.815338 (+ 39us) server_negotiation.cc:407] Connection header received
0927 02:53:07.815450 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.815451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.815464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.815526 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.815723 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.816407 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.817400 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.817780 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.817847 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.817859 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.818307 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.818360 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:53:07.818507 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.815336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49369 (local address 127.0.98.3:35369)
0927 02:53:07.815374 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.815376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.815414 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:53:07.815439 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.815440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.815450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.815500 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:07.815655 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.816394 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.817805 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.818185 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.818243 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.818252 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.818404 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.818440 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:53:07.818629 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf3b3ca6cbd4446ae8b059a059fdec1" candidate_uuid: "34282bd6edb14a04a35651b4b019dafd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e02f9699042f6b406d52d0fc3e664" is_pre_election: true
I20260927 02:53:07.818718 8165 raft_consensus.cc:2454] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34282bd6edb14a04a35651b4b019dafd in term 0.
I20260927 02:53:07.818922 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.815145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35369 (local address 127.0.98.1:49369)
0927 02:53:07.815324 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:07.815333 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:07.815372 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:07.815504 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:07.815505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:07.815507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:07.815587 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.815590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.816491 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.816492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.817710 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:07.817713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.818255 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.818256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:07.818325 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.818329 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:07.818744 (+ 415us) client_negotiation.cc:240] Negotiation successful
0927 02:53:07.818827 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":28,"thread_start_us":70,"threads_started":1}
I20260927 02:53:07.818912 7957 leader_election.cc:302] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34282bd6edb14a04a35651b4b019dafd, c84e02f9699042f6b406d52d0fc3e664; no voters:
I20260927 02:53:07.819044 8385 raft_consensus.cc:2781] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:53:07.819072 8385 raft_consensus.cc:482] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:53:07.819103 8385 raft_consensus.cc:3037] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.819203 8305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf3b3ca6cbd4446ae8b059a059fdec1" candidate_uuid: "34282bd6edb14a04a35651b4b019dafd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" is_pre_election: true
I20260927 02:53:07.819284 8305 raft_consensus.cc:2454] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34282bd6edb14a04a35651b4b019dafd in term 0.
I20260927 02:53:07.819849 8385 raft_consensus.cc:504] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.819967 8385 leader_election.cc:288] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [CANDIDATE]: Term 1 election: Requested vote from peers c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439), e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369)
I20260927 02:53:07.820127 8305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf3b3ca6cbd4446ae8b059a059fdec1" candidate_uuid: "34282bd6edb14a04a35651b4b019dafd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d"
I20260927 02:53:07.820125 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf3b3ca6cbd4446ae8b059a059fdec1" candidate_uuid: "34282bd6edb14a04a35651b4b019dafd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e02f9699042f6b406d52d0fc3e664"
I20260927 02:53:07.820201 8305 raft_consensus.cc:3037] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.820201 8165 raft_consensus.cc:3037] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:07.820914 8165 raft_consensus.cc:2454] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34282bd6edb14a04a35651b4b019dafd in term 1.
I20260927 02:53:07.820914 8305 raft_consensus.cc:2454] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34282bd6edb14a04a35651b4b019dafd in term 1.
I20260927 02:53:07.821084 7957 leader_election.cc:302] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34282bd6edb14a04a35651b4b019dafd, c84e02f9699042f6b406d52d0fc3e664; no voters:
I20260927 02:53:07.821197 8385 raft_consensus.cc:2781] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:07.821341 8385 raft_consensus.cc:686] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 1 LEADER]: Becoming Leader. State: Replica: 34282bd6edb14a04a35651b4b019dafd, State: Running, Role: LEADER
I20260927 02:53:07.821413 8385 consensus_queue.cc:237] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.822089 7868 catalog_manager.cc:5495] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd reported cstate change: term changed from 0 to 1, leader changed from <none> to 34282bd6edb14a04a35651b4b019dafd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "34282bd6edb14a04a35651b4b019dafd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } health_report { overall_health: UNKNOWN } } }
I20260927 02:53:07.828809 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:07.826235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.0.98.1:34077)
0927 02:53:07.826279 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:07.826281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:07.826341 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:53:07.826424 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:07.826425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:07.826444 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:07.826511 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:07.826646 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.827160 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.828204 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:07.828376 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:07.828415 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:07.828559 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:07.828649 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:07.828695 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:07.828737 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:53:07.828750 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:53:07.829677 8305 raft_consensus.cc:1261] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Refusing update from remote peer 34282bd6edb14a04a35651b4b019dafd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:07.829843 8165 raft_consensus.cc:1261] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Refusing update from remote peer 34282bd6edb14a04a35651b4b019dafd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:07.830088 8451 consensus_queue.cc:1035] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:07.830216 8385 consensus_queue.cc:1035] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:07.833156 7867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:07.833232 7867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:07.833730 7867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: Sending DeleteTablet for 3 replicas of tablet 2bf3b3ca6cbd4446ae8b059a059fdec1
I20260927 02:53:07.833940 8284 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260927 02:53:07.833985 8005 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57760
I20260927 02:53:07.834034 8145 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46740
I20260927 02:53:07.834110 8402 tablet_replica.cc:330] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: stopping tablet replica
I20260927 02:53:07.834110 8404 tablet_replica.cc:330] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: stopping tablet replica
I20260927 02:53:07.834153 8402 raft_consensus.cc:2229] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:07.834157 8404 raft_consensus.cc:2229] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:07.834188 8402 raft_consensus.cc:2258] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.834195 8404 raft_consensus.cc:2258] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.834247 8403 tablet_replica.cc:330] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: stopping tablet replica
I20260927 02:53:07.834286 8403 raft_consensus.cc:2229] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:53:07.834347 8403 raft_consensus.cc:2258] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:07.834627 8403 ts_tablet_manager.cc:1905] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.834875 8402 ts_tablet_manager.cc:1905] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.836119 8403 ts_tablet_manager.cc:1918] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:07.836169 8403 log.cc:1196] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/2bf3b3ca6cbd4446ae8b059a059fdec1
I20260927 02:53:07.836350 8403 ts_tablet_manager.cc:1939] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P 34282bd6edb14a04a35651b4b019dafd: Deleting consensus metadata
I20260927 02:53:07.836460 7867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:53:07.836586 7867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:53:07.836615 7853 catalog_manager.cc:4841] TS 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077): tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 (table test-table [id=cadbfdd4c7744727913adffe013da97f]) successfully deleted
I20260927 02:53:07.837568 8402 ts_tablet_manager.cc:1918] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:07.837620 8402 log.cc:1196] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/2bf3b3ca6cbd4446ae8b059a059fdec1
I20260927 02:53:07.837728 8005 tablet_service.cc:1467] Processing CreateTablet for tablet 468da25d474a4afebd894502f5980b3f (DEFAULT_TABLE table=test-table [id=d198c3c7559145ceb0516c4c4383c64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.837803 8402 ts_tablet_manager.cc:1939] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting consensus metadata
I20260927 02:53:07.837808 8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468da25d474a4afebd894502f5980b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.838081 8145 tablet_service.cc:1467] Processing CreateTablet for tablet 468da25d474a4afebd894502f5980b3f (DEFAULT_TABLE table=test-table [id=d198c3c7559145ceb0516c4c4383c64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.838173 8145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468da25d474a4afebd894502f5980b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.838809 8380 tablet_bootstrap.cc:492] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd: Bootstrap starting.
I20260927 02:53:07.839175 8380 tablet_bootstrap.cc:654] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.839387 8379 tablet_bootstrap.cc:492] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: Bootstrap starting.
I20260927 02:53:07.839636 8379 tablet_bootstrap.cc:654] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.839911 7852 catalog_manager.cc:4841] TS e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369): tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 (table test-table [id=cadbfdd4c7744727913adffe013da97f]) successfully deleted
I20260927 02:53:07.840054 8379 tablet_bootstrap.cc:492] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: No bootstrap required, opened a new log
I20260927 02:53:07.840054 8284 tablet_service.cc:1467] Processing CreateTablet for tablet 468da25d474a4afebd894502f5980b3f (DEFAULT_TABLE table=test-table [id=d198c3c7559145ceb0516c4c4383c64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:07.840092 8379 ts_tablet_manager.cc:1397] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:53:07.840134 8284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468da25d474a4afebd894502f5980b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:07.840186 8379 raft_consensus.cc:348] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.840245 8379 raft_consensus.cc:374] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.840262 8379 raft_consensus.cc:729] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e02f9699042f6b406d52d0fc3e664, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.840301 8379 consensus_queue.cc:260] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.840376 8379 ts_tablet_manager.cc:1428] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.841094 8384 tablet_bootstrap.cc:492] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: Bootstrap starting.
I20260927 02:53:07.841377 8384 tablet_bootstrap.cc:654] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:07.842037 8404 ts_tablet_manager.cc:1905] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:07.843190 8404 ts_tablet_manager.cc:1918] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:07.843241 8404 log.cc:1196] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/2bf3b3ca6cbd4446ae8b059a059fdec1
I20260927 02:53:07.843422 8404 ts_tablet_manager.cc:1939] T 2bf3b3ca6cbd4446ae8b059a059fdec1 P c84e02f9699042f6b406d52d0fc3e664: Deleting consensus metadata
I20260927 02:53:07.843665 8384 tablet_bootstrap.cc:492] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: No bootstrap required, opened a new log
I20260927 02:53:07.843703 8384 ts_tablet_manager.cc:1397] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260927 02:53:07.843766 8380 tablet_bootstrap.cc:492] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd: No bootstrap required, opened a new log
I20260927 02:53:07.843801 8380 ts_tablet_manager.cc:1397] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 02:53:07.843806 8384 raft_consensus.cc:348] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.843866 8384 raft_consensus.cc:374] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.843883 8384 raft_consensus.cc:729] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e27d14e74a4d42a5dd1a974da19f9d, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.843921 8384 consensus_queue.cc:260] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.843922 8380 raft_consensus.cc:348] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.843990 8380 raft_consensus.cc:374] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:07.844012 8380 raft_consensus.cc:729] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34282bd6edb14a04a35651b4b019dafd, State: Initialized, Role: FOLLOWER
I20260927 02:53:07.844007 8384 ts_tablet_manager.cc:1428] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.844049 8380 consensus_queue.cc:260] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:07.844121 8380 ts_tablet_manager.cc:1428] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:07.844326 7853 catalog_manager.cc:4841] TS c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439): tablet 2bf3b3ca6cbd4446ae8b059a059fdec1 (table test-table [id=cadbfdd4c7744727913adffe013da97f]) successfully deleted
I20260927 02:53:08.060905 8432 raft_consensus.cc:482] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:53:08.060998 8432 raft_consensus.cc:504] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:08.061151 8432 leader_election.cc:288] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077), c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439)
I20260927 02:53:08.061412 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468da25d474a4afebd894502f5980b3f" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34282bd6edb14a04a35651b4b019dafd" is_pre_election: true
I20260927 02:53:08.061434 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468da25d474a4afebd894502f5980b3f" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e02f9699042f6b406d52d0fc3e664" is_pre_election: true
I20260927 02:53:08.061517 8025 raft_consensus.cc:2454] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 0.
I20260927 02:53:08.061518 8165 raft_consensus.cc:2454] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 0.
I20260927 02:53:08.061668 8237 leader_election.cc:302] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34282bd6edb14a04a35651b4b019dafd, e2e27d14e74a4d42a5dd1a974da19f9d; no voters:
I20260927 02:53:08.061748 8432 raft_consensus.cc:2781] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:53:08.061798 8432 raft_consensus.cc:482] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:53:08.061823 8432 raft_consensus.cc:3037] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:08.062487 8432 raft_consensus.cc:504] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:08.062587 8432 leader_election.cc:288] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 election: Requested vote from peers 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077), c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439)
I20260927 02:53:08.062731 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468da25d474a4afebd894502f5980b3f" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34282bd6edb14a04a35651b4b019dafd"
I20260927 02:53:08.062789 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468da25d474a4afebd894502f5980b3f" candidate_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e02f9699042f6b406d52d0fc3e664"
I20260927 02:53:08.062848 8165 raft_consensus.cc:3037] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:08.062851 8025 raft_consensus.cc:3037] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:08.063599 8165 raft_consensus.cc:2454] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 1.
I20260927 02:53:08.063599 8025 raft_consensus.cc:2454] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e27d14e74a4d42a5dd1a974da19f9d in term 1.
I20260927 02:53:08.063781 8237 leader_election.cc:302] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34282bd6edb14a04a35651b4b019dafd, e2e27d14e74a4d42a5dd1a974da19f9d; no voters:
I20260927 02:53:08.063928 8432 raft_consensus.cc:2781] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:08.063992 8432 raft_consensus.cc:686] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 LEADER]: Becoming Leader. State: Replica: e2e27d14e74a4d42a5dd1a974da19f9d, State: Running, Role: LEADER
I20260927 02:53:08.064042 8432 consensus_queue.cc:237] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } }
I20260927 02:53:08.064553 7867 catalog_manager.cc:5495] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2e27d14e74a4d42a5dd1a974da19f9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35369 } health_report { overall_health: HEALTHY } } }
I20260927 02:53:08.126742 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:08.124076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.0.98.3:35369)
0927 02:53:08.124118 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:08.124120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:08.124132 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:53:08.124203 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:08.124204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:08.124215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:08.124262 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:08.124384 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:08.125104 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:08.126035 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:08.126331 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:08.126413 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:08.126532 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:08.126568 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:08.126609 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:08.126643 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 02:53:08.126663 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:53:08.128769 8025 raft_consensus.cc:1261] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Refusing update from remote peer e2e27d14e74a4d42a5dd1a974da19f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:08.128893 8165 raft_consensus.cc:1261] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Refusing update from remote peer e2e27d14e74a4d42a5dd1a974da19f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:08.128989 8409 consensus_queue.cc:1035] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "34282bd6edb14a04a35651b4b019dafd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:08.129119 8409 consensus_queue.cc:1035] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e02f9699042f6b406d52d0fc3e664" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:08.138581 7868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:08.138705 7868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:08.139295 7868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e23abef7edc441da1be047ac15925a9: Sending DeleteTablet for 3 replicas of tablet 468da25d474a4afebd894502f5980b3f
I20260927 02:53:08.139505 8005 tablet_service.cc:1514] Processing DeleteTablet for tablet 468da25d474a4afebd894502f5980b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57760
I20260927 02:53:08.139544 8145 tablet_service.cc:1514] Processing DeleteTablet for tablet 468da25d474a4afebd894502f5980b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46740
I20260927 02:53:08.139605 8403 tablet_replica.cc:330] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd: stopping tablet replica
I20260927 02:53:08.139626 8404 tablet_replica.cc:330] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: stopping tablet replica
I20260927 02:53:08.139655 8403 raft_consensus.cc:2229] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:08.139671 8404 raft_consensus.cc:2229] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:08.139694 8403 raft_consensus.cc:2258] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:08.139731 8404 raft_consensus.cc:2258] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:08.139976 8284 tablet_service.cc:1514] Processing DeleteTablet for tablet 468da25d474a4afebd894502f5980b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45756
I20260927 02:53:08.140054 8402 tablet_replica.cc:330] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: stopping tablet replica
I20260927 02:53:08.140057 8403 ts_tablet_manager.cc:1905] T 468da25d474a4afebd894502f5980b3f P 34282bd6edb14a04a35651b4b019dafd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:08.140100 8402 raft_consensus.cc:2229] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:53:08.140153 8402 raft_consensus.cc:2258] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:08.140472 8402 ts_tablet_manager.cc:1905] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:08.140722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 7940
I20260927 02:53:08.141377 8404 ts_tablet_manager.cc:1905] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:08.142321 8402 ts_tablet_manager.cc:1918] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:53:08.142369 8402 log.cc:1196] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/468da25d474a4afebd894502f5980b3f
I20260927 02:53:08.142555 8402 ts_tablet_manager.cc:1939] T 468da25d474a4afebd894502f5980b3f P e2e27d14e74a4d42a5dd1a974da19f9d: Deleting consensus metadata
I20260927 02:53:08.142807 8404 ts_tablet_manager.cc:1918] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:53:08.142875 8404 log.cc:1196] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/468da25d474a4afebd894502f5980b3f
I20260927 02:53:08.143062 8404 ts_tablet_manager.cc:1939] T 468da25d474a4afebd894502f5980b3f P c84e02f9699042f6b406d52d0fc3e664: Deleting consensus metadata
I20260927 02:53:08.143062 7852 catalog_manager.cc:4841] TS e2e27d14e74a4d42a5dd1a974da19f9d (127.0.98.3:35369): tablet 468da25d474a4afebd894502f5980b3f (table test-table [id=d198c3c7559145ceb0516c4c4383c64f]) successfully deleted
I20260927 02:53:08.143400 7853 catalog_manager.cc:4841] TS c84e02f9699042f6b406d52d0fc3e664 (127.0.98.2:44439): tablet 468da25d474a4afebd894502f5980b3f (table test-table [id=d198c3c7559145ceb0516c4c4383c64f]) successfully deleted
W20260927 02:53:08.146529 7853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34077 (error 108)
I20260927 02:53:08.146801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 8080
I20260927 02:53:08.149415 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:08.149221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34077 (local address 127.0.0.1:57782)
0927 02:53:08.149313 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:08.149369 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":41,"threads_started":1}
W20260927 02:53:08.152505 7853 catalog_manager.cc:4584] TS 34282bd6edb14a04a35651b4b019dafd (127.0.98.1:34077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 468da25d474a4afebd894502f5980b3f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34077: connect: Connection refused (error 111)
I20260927 02:53:08.152724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 8220
I20260927 02:53:08.157673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 7836
2026-09-27T02:53:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2651 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:53:08 dist-test-slave-mfx9 krb5kdc[8467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:53:08 dist-test-slave-mfx9 krb5kdc[8467](info): set up 2 sockets
Sep 27 02:53:08 dist-test-slave-mfx9 krb5kdc[8467](info): commencing operation
krb5kdc: starting...
W20260927 02:53:09.107004 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:53:09 dist-test-slave-mfx9 krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:53:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:53:09.219607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35499
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:09.296489 8480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:09.296679 8480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:09.296698 8480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:09.297766 8480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:09.297799 8480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:09.297811 8480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:09.297822 8480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:09.297832 8480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:09.299289 8480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:09.299481 8480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:09.299645 8480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:09.304171 8488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:09.304176 8486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:09.304205 8485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:09.304342 8480 server_base.cc:1034] running on GCE node
I20260927 02:53:09.304708 8480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:09.304956 8480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:09.306134 8480 hybrid_clock.cc:648] HybridClock initialized: now 1790477589306116 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T02:53:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1148 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:53:09 dist-test-slave-mfx9 krb5kdc[8493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:53:09 dist-test-slave-mfx9 krb5kdc[8493](info): set up 2 sockets
Sep 27 02:53:09 dist-test-slave-mfx9 krb5kdc[8493](info): commencing operation
krb5kdc: starting...
W20260927 02:53:10.247020 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:53:10.354244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36627
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:10.431005 8506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:10.431200 8506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:10.431219 8506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:10.432341 8506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:10.432379 8506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:10.432390 8506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:10.432401 8506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:10.432411 8506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:10.433848 8506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:10.434034 8506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:10.434199 8506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:10.438685 8512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:10.438714 8511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:10.438735 8506 server_base.cc:1034] running on GCE node
W20260927 02:53:10.438714 8514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:10.439159 8506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:10.439422 8506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:10.440562 8506 hybrid_clock.cc:648] HybridClock initialized: now 1790477590440542 us; error 36 us; skew 500 ppm
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:10.442886 8506 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:53:10.443276 8506 webserver.cc:460] Webserver started at http://127.0.98.62:38439/ using document root <none> and password file <none>
I20260927 02:53:10.443465 8506 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:10.443512 8506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:10.443630 8506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:10.444607 8506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "ee7c2a250a584310bccba79466aa0171"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "d412924cabcb5a0e6d00aa70deb62662"
server_key_iv: "ea738f742893e2e775698f0ac9300c28"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.444924 8506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee7c2a250a584310bccba79466aa0171"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "d412924cabcb5a0e6d00aa70deb62662"
server_key_iv: "ea738f742893e2e775698f0ac9300c28"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.446085 8506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:10.446808 8521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.446957 8506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:10.447039 8506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "ee7c2a250a584310bccba79466aa0171"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "d412924cabcb5a0e6d00aa70deb62662"
server_key_iv: "ea738f742893e2e775698f0ac9300c28"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.447103 8506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:10.467793 8506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:10.469676 8506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:10.469815 8506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:10.473413 8506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36627
I20260927 02:53:10.473451 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36627 every 8 connection(s)
I20260927 02:53:10.473904 8506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:53:10.474537 8574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:10.476747 8574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Bootstrap starting.
I20260927 02:53:10.477289 8574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:10.477540 8574 log.cc:824] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:10.478121 8574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: No bootstrap required, opened a new log
I20260927 02:53:10.478225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 8506
I20260927 02:53:10.478323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:53:10.478485 392 external_mini_cluster.cc:1428] Setting key fe38b86681e17024472a805af49c0c48
I20260927 02:53:10.480557 8574 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260927 02:53:10.480719 8574 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:10.480753 8574 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7c2a250a584310bccba79466aa0171, State: Initialized, Role: FOLLOWER
I20260927 02:53:10.480880 8574 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260927 02:53:10.480947 8574 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:53:10.480967 8574 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:53:10.481010 8574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:10.481557 8574 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260927 02:53:10.481657 8574 leader_election.cc:302] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee7c2a250a584310bccba79466aa0171; no voters:
I20260927 02:53:10.481794 8574 leader_election.cc:288] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:53:10.481870 8579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:10.482009 8579 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 1 LEADER]: Becoming Leader. State: Replica: ee7c2a250a584310bccba79466aa0171, State: Running, Role: LEADER
I20260927 02:53:10.482043 8574 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:10.482095 8579 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260927 02:53:10.482404 8579 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee7c2a250a584310bccba79466aa0171. Latest consensus state: current_term: 1 leader_uuid: "ee7c2a250a584310bccba79466aa0171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } } }
I20260927 02:53:10.482496 8579 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:10.482414 8580 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee7c2a250a584310bccba79466aa0171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } } }
I20260927 02:53:10.482612 8580 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:10.482697 8586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:10.483188 8586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:10.487841 8586 catalog_manager.cc:1347] Generated new cluster ID: 7009d29a318d4b7393e867f153afd3b2
I20260927 02:53:10.487895 8586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:53:10.489861 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:10.478940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.0.98.62:36627)
0927 02:53:10.479098 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:10.479107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:10.479140 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:53:10.480521 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:10.480541 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:10.480604 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:10.480750 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:10.481380 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.482705 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.483759 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.484247 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.484370 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:10.486715 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:10.486718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:10.486719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:10.486729 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:10.488455 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:10.488770 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:10.488773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:10.488774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:10.488870 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:10.489044 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:10.489044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:10.489045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:10.489267 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:10.489424 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:10.489492 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:53:10.489711 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260927 02:53:10.495991 8586 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:53:10.496462 8586 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:10.502616 8586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Generated new TSK 0
I20260927 02:53:10.502813 8586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:53:10.537257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:10.637049 8600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:10.637238 8600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:10.637259 8600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:10.638378 8600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:10.638417 8600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:10.638444 8600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:10.639915 8600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:10.640115 8600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:10.640349 8600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:10.645088 8608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:10.645188 8605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:10.645195 8606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:10.645395 8600 server_base.cc:1034] running on GCE node
I20260927 02:53:10.645599 8600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:10.645809 8600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:10.646940 8600 hybrid_clock.cc:648] HybridClock initialized: now 1790477590646915 us; error 46 us; skew 500 ppm
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:10.649282 8600 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:53:10.649698 8600 webserver.cc:460] Webserver started at http://127.0.98.1:39437/ using document root <none> and password file <none>
I20260927 02:53:10.649892 8600 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:10.649937 8600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:10.650058 8600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:10.651122 8600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "49305a8b31a441839dd1b773a6861f84"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "43b7ef64533897571f546cd420ca34c9"
server_key_iv: "bec7189fab0ba404475b7daf0383e3c3"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.651425 8600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "49305a8b31a441839dd1b773a6861f84"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "43b7ef64533897571f546cd420ca34c9"
server_key_iv: "bec7189fab0ba404475b7daf0383e3c3"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.652575 8600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:53:10.653194 8615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.653340 8600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:10.653414 8600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "49305a8b31a441839dd1b773a6861f84"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "43b7ef64533897571f546cd420ca34c9"
server_key_iv: "bec7189fab0ba404475b7daf0383e3c3"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.653469 8600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:10.663547 8600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:10.665233 8600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:10.665354 8600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:10.665546 8600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:10.665812 8600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:10.665841 8600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.665868 8600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:10.665885 8600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.671060 8600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38501
I20260927 02:53:10.671123 8728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38501 every 8 connection(s)
I20260927 02:53:10.671377 8600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:53:10.672107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 8600
I20260927 02:53:10.672212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:53:10.672397 392 external_mini_cluster.cc:1428] Setting key 699dc54e7912bd7d357e46fe0ae01ee3
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:53:10.681546 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:10.672048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44949 (local address 127.0.98.62:36627)
0927 02:53:10.672110 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:10.672112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:10.672249 (+ 137us) server_negotiation.cc:407] Connection header received
0927 02:53:10.673268 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:10.673269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:10.673281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:10.673345 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:10.673999 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.674694 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.676113 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.676885 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.676961 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:10.679204 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:10.679205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:10.679205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:10.679213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:10.680457 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:10.680822 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:10.680823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:10.680823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:10.680869 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:10.681025 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:10.681025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:10.681025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:10.681116 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:10.681160 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:10.681405 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 02:53:10.681473 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:53:10.681667 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:10.672001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:44949)
0927 02:53:10.672143 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:10.672181 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:10.673180 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:10.673406 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:10.673410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:10.673708 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:10.673896 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:10.673901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.674815 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.674816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:10.676025 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:10.676028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.676968 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.676968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:10.677077 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:10.677539 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:10.677546 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:10.679106 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:10.680541 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:10.680548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:10.680550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:10.680760 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:10.680906 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:10.680906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:10.680906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:10.680986 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:10.681161 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:10.681166 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:10.681300 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 02:53:10.681380 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:53:10.681494 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260927 02:53:10.682080 8729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36627
I20260927 02:53:10.682168 8729 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:10.682372 8729 heartbeater.cc:511] Master 127.0.98.62:36627 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:53:10.682893 8538 ts_manager.cc:194] Registered new tserver with Master: 49305a8b31a441839dd1b773a6861f84 (127.0.98.1:38501)
I20260927 02:53:10.683531 8538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:53:10.696779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:10.773736 8734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:10.773928 8734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:10.773957 8734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:10.775112 8734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:10.775153 8734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:10.775187 8734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:10.776630 8734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:10.776852 8734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:10.777055 8734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:10.781713 8740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:10.781752 8742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:10.781723 8739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:10.781941 8734 server_base.cc:1034] running on GCE node
I20260927 02:53:10.782155 8734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:10.782366 8734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:10.783502 8734 hybrid_clock.cc:648] HybridClock initialized: now 1790477590783485 us; error 30 us; skew 500 ppm
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:10.785817 8734 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:53:10.786244 8734 webserver.cc:460] Webserver started at http://127.0.98.2:43629/ using document root <none> and password file <none>
I20260927 02:53:10.786437 8734 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:10.786482 8734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:10.786593 8734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:10.787627 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae20c48e53d145099fa5d18b5c0a3ab1"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "72dbc9eb73b6c56e36432bc7076bb9e6"
server_key_iv: "cc5261cc40c7bc89b6282b3c0bf56f5d"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.787945 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae20c48e53d145099fa5d18b5c0a3ab1"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "72dbc9eb73b6c56e36432bc7076bb9e6"
server_key_iv: "cc5261cc40c7bc89b6282b3c0bf56f5d"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.789106 8734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:53:10.789755 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.789901 8734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:10.789976 8734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "ae20c48e53d145099fa5d18b5c0a3ab1"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "72dbc9eb73b6c56e36432bc7076bb9e6"
server_key_iv: "cc5261cc40c7bc89b6282b3c0bf56f5d"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.790045 8734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:10.812755 8734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:10.814682 8734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:10.814845 8734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:10.815069 8734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:10.815366 8734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:10.815397 8734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.815428 8734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:10.815449 8734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.820585 8734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41993
I20260927 02:53:10.820626 8862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41993 every 8 connection(s)
I20260927 02:53:10.820919 8734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:53:10.821053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 8734
I20260927 02:53:10.821153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:53:10.821363 392 external_mini_cluster.cc:1428] Setting key 58f1e3c1599cef441c6901ed2d4193cc
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:53:10.830205 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:10.821620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60467 (local address 127.0.98.62:36627)
0927 02:53:10.821679 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:10.821681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:10.821882 (+ 201us) server_negotiation.cc:407] Connection header received
0927 02:53:10.823060 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:10.823061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:10.823070 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:10.823108 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:10.823834 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.824323 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.825379 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.825586 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.825640 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:10.827680 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:10.827682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:10.827682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:10.827690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:10.829144 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:10.829476 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:10.829476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:10.829476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:10.829521 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:10.829700 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:10.829701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:10.829701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:10.829819 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:10.829870 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:10.830080 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 02:53:10.830139 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:53:10.830371 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:10.821594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:60467)
0927 02:53:10.821751 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:10.821798 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:10.822953 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:10.823180 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:10.823184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:10.823531 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:10.823742 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:10.823747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.824402 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.824403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:10.825306 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:10.825308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.825651 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.825652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:10.825778 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:10.826246 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:10.826255 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:10.827600 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:10.829220 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:10.829225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:10.829227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:10.829422 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:10.829556 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:10.829557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:10.829557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:10.829655 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:10.829867 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:10.829870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:10.829985 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 02:53:10.830061 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:53:10.830172 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260927 02:53:10.830807 8863 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36627
I20260927 02:53:10.830899 8863 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:53:10.831076 8863 heartbeater.cc:511] Master 127.0.98.62:36627 requested a full tablet report, sending...
I20260927 02:53:10.831434 8538 ts_manager.cc:194] Registered new tserver with Master: ae20c48e53d145099fa5d18b5c0a3ab1 (127.0.98.2:41993)
I20260927 02:53:10.831784 8538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:53:10.843386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:10.920492 8868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:10.920703 8868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:10.920727 8868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:10.921832 8868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:10.921871 8868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:10.921900 8868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:10.923354 8868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:10.923554 8868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:10.923753 8868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:10.928243 8874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:10.928357 8868 server_base.cc:1034] running on GCE node
W20260927 02:53:10.928262 8873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:10.928251 8876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:10.928659 8868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:10.928869 8868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:10.930001 8868 hybrid_clock.cc:648] HybridClock initialized: now 1790477590929985 us; error 25 us; skew 500 ppm
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:10.932339 8868 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:53:10.932715 8868 webserver.cc:460] Webserver started at http://127.0.98.3:35115/ using document root <none> and password file <none>
I20260927 02:53:10.932896 8868 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:10.932941 8868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:10.933066 8868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:10.934005 8868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "96e1d0db893b413cac8423bee1c2fdc5"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "c82e3e45170c84e3e68ac09661a4b7ab"
server_key_iv: "e324bbaeaacca2c5c4c8b78316b6275b"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.934336 8868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "96e1d0db893b413cac8423bee1c2fdc5"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "c82e3e45170c84e3e68ac09661a4b7ab"
server_key_iv: "e324bbaeaacca2c5c4c8b78316b6275b"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.935547 8868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:10.936149 8883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.936300 8868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:10.936375 8868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "96e1d0db893b413cac8423bee1c2fdc5"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "c82e3e45170c84e3e68ac09661a4b7ab"
server_key_iv: "e324bbaeaacca2c5c4c8b78316b6275b"
server_key_version: "encryptionkey@0"
I20260927 02:53:10.936430 8868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:10.977985 8868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:10.980017 8868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:10.980144 8868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:10.980352 8868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:10.980639 8868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:10.980672 8868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.980701 8868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:10.980724 8868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:10.986140 8868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43329
I20260927 02:53:10.986191 8996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43329 every 8 connection(s)
I20260927 02:53:10.986472 8868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:53:10.987738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 8868
I20260927 02:53:10.987851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:53:10.988037 392 external_mini_cluster.cc:1428] Setting key e204146f3d26aec9cca0eabc4b8e9d81
Sep 27 02:53:10 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:10.995360 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:10.987089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40659 (local address 127.0.98.62:36627)
0927 02:53:10.987159 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:10.987161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:10.987329 (+ 168us) server_negotiation.cc:407] Connection header received
0927 02:53:10.988497 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:10.988499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:10.988511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:10.988562 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:10.989191 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.989711 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.990650 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.990899 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.990955 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:10.992876 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:10.992877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:10.992878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:10.992885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:10.994096 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:10.994446 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:10.994448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:10.994448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:10.994511 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:10.994711 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:10.994712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:10.994713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:10.994853 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:10.994908 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:10.995188 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:53:10.995264 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:53:10.995482 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:10.987078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:40659)
0927 02:53:10.987216 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:10.987251 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:10.988397 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:10.988637 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:10.988641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:10.988904 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:10.989111 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:10.989116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.989787 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.989787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:10.990571 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:10.990574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:10.990963 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:10.990965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:10.991075 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:10.991511 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:10.991519 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:10.992791 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:10.994169 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:10.994175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:10.994178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:10.994382 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:10.994563 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:10.994564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:10.994564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:10.994658 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:10.994904 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:10.994910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:10.995039 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 02:53:10.995129 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:53:10.995274 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260927 02:53:10.995913 8997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36627
I20260927 02:53:10.995999 8997 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:10.996143 8997 heartbeater.cc:511] Master 127.0.98.62:36627 requested a full tablet report, sending...
I20260927 02:53:10.996484 8538 ts_manager.cc:194] Registered new tserver with Master: 96e1d0db893b413cac8423bee1c2fdc5 (127.0.98.3:43329)
I20260927 02:53:10.996804 8538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40659
I20260927 02:53:10.999604 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:53:11.061362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 8600
I20260927 02:53:11.065606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 8734
I20260927 02:53:11.069099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 8868
I20260927 02:53:11.072376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 8506
I20260927 02:53:11.076423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36627
--webserver_interface=127.0.98.62
--webserver_port=38439
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:11.154079 9006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:11.154305 9006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:11.154335 9006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:11.155524 9006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:11.155562 9006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:11.155581 9006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:11.155596 9006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:11.155611 9006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:11.157124 9006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:11.157356 9006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:11.157549 9006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:11.162034 9012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:11.161966 9011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:11.162154 9014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:11.162335 9006 server_base.cc:1034] running on GCE node
I20260927 02:53:11.162577 9006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:11.162879 9006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:11.164048 9006 hybrid_clock.cc:648] HybridClock initialized: now 1790477591164022 us; error 37 us; skew 500 ppm
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:11.167086 9006 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:53:11.167522 9006 webserver.cc:460] Webserver started at http://127.0.98.62:38439/ using document root <none> and password file <none>
I20260927 02:53:11.167723 9006 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:11.167778 9006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:11.169104 9006 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:53:11.169782 9021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.169948 9006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:53:11.170032 9006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "ee7c2a250a584310bccba79466aa0171"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "d412924cabcb5a0e6d00aa70deb62662"
server_key_iv: "ea738f742893e2e775698f0ac9300c28"
server_key_version: "encryptionkey@0"
I20260927 02:53:11.170387 9006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:11.186704 9006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:11.188843 9006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:11.188983 9006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:11.192626 9006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36627
I20260927 02:53:11.192986 9006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:53:11.192973 9073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36627 every 8 connection(s)
I20260927 02:53:11.194278 9074 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:53:11.194927 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Bootstrap starting.
I20260927 02:53:11.200882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 9006
I20260927 02:53:11.201136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38501
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36627
--webserver_port=39437
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:53:11.202948 9074 log.cc:824] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:11.204527 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:53:11.204774 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Bootstrap complete.
I20260927 02:53:11.206833 9074 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260927 02:53:11.207065 9074 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7c2a250a584310bccba79466aa0171, State: Initialized, Role: FOLLOWER
I20260927 02:53:11.207206 9074 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260927 02:53:11.207264 9074 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:53:11.207304 9074 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:53:11.207347 9074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:53:11.208273 9074 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260927 02:53:11.208357 9074 leader_election.cc:302] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee7c2a250a584310bccba79466aa0171; no voters:
I20260927 02:53:11.208503 9074 leader_election.cc:288] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:53:11.208549 9078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:53:11.208657 9078 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [term 2 LEADER]: Becoming Leader. State: Replica: ee7c2a250a584310bccba79466aa0171, State: Running, Role: LEADER
I20260927 02:53:11.208738 9078 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260927 02:53:11.208763 9074 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:11.208997 9078 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee7c2a250a584310bccba79466aa0171. Latest consensus state: current_term: 2 leader_uuid: "ee7c2a250a584310bccba79466aa0171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } } }
I20260927 02:53:11.209074 9078 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:11.209000 9079 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ee7c2a250a584310bccba79466aa0171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c2a250a584310bccba79466aa0171" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } } }
I20260927 02:53:11.209235 9079 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:11.209267 9082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:11.209725 9082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:53:11.209862 9082 catalog_manager.cc:1259] Loaded cluster ID: 7009d29a318d4b7393e867f153afd3b2
I20260927 02:53:11.209900 9082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:53:11.211072 9082 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:11.211232 9082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ee7c2a250a584310bccba79466aa0171: Loaded TSK: 0
I20260927 02:53:11.211439 9082 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:53:11.278996 9076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:11.279183 9076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:11.279213 9076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:11.280377 9076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:11.280418 9076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:11.280452 9076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:11.281862 9076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39437
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:11.282074 9076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:11.282270 9076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:11.286980 9101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:11.286980 9103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:11.286962 9100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:11.287213 9076 server_base.cc:1034] running on GCE node
I20260927 02:53:11.287421 9076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:11.287696 9076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:11.288875 9076 hybrid_clock.cc:648] HybridClock initialized: now 1790477591288861 us; error 27 us; skew 500 ppm
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:11.291692 9076 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:53:11.292098 9076 webserver.cc:460] Webserver started at http://127.0.98.1:39437/ using document root <none> and password file <none>
I20260927 02:53:11.292302 9076 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:11.292354 9076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:11.293670 9076 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:11.294221 9110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.294354 9076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.294420 9076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "49305a8b31a441839dd1b773a6861f84"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "43b7ef64533897571f546cd420ca34c9"
server_key_iv: "bec7189fab0ba404475b7daf0383e3c3"
server_key_version: "encryptionkey@0"
I20260927 02:53:11.294739 9076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:11.326064 9076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:11.328008 9076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:11.328123 9076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:11.328341 9076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:11.328619 9076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:11.328650 9076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.328671 9076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:11.328694 9076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.333909 9076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38501
I20260927 02:53:11.333973 9223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38501 every 8 connection(s)
I20260927 02:53:11.334260 9076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:53:11.335675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 9076
I20260927 02:53:11.335955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41993
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36627
--webserver_port=43629
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.342563 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.334932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:34939)
0927 02:53:11.335088 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.335129 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.335575 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.336086 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.336090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.336646 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.336826 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.336830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.337860 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.337861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.339020 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.339023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.339609 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.339610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.339748 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.340182 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.340193 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.342366 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260927 02:53:11.342747 9224 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36627 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36627: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260927 02:53:11.342634 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.334954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34939 (local address 127.0.98.62:36627)
0927 02:53:11.335124 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.335134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.335200 (+ 66us) server_negotiation.cc:407] Connection header received
0927 02:53:11.335770 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.335778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.335835 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.336038 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.336941 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.337774 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.339103 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.339526 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.339613 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.342495 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34939: BlockingRecv error: recv got EOF from 127.0.98.1:34939 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.347249 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.342978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:55017)
0927 02:53:11.343026 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.343037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.343075 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.343202 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.343203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.343302 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.343346 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.343347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.344167 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.344168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.344660 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.344662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.345070 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.345071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.345111 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.345458 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.345462 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.347201 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:53:11.347353 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.343030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55017 (local address 127.0.98.62:36627)
0927 02:53:11.343067 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.343069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.343077 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:11.343133 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.343134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.343145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.343196 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.343403 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.344096 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.344731 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.345007 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.345075 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.347296 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55017: BlockingRecv error: recv got EOF from 127.0.98.1:55017 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.351805 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.347524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:54909)
0927 02:53:11.347559 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.347568 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.347599 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.347846 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.347847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.347955 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.347982 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.347983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.348859 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.348860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.349376 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.349378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.349802 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.349803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.349842 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.350178 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.350182 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.351762 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:53:11.351909 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.347579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54909 (local address 127.0.98.62:36627)
0927 02:53:11.347696 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.347697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.347705 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:11.347724 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.347725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.347735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.347790 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.348043 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.348770 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.349437 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.349729 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.349787 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.351852 (+ 2065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54909: BlockingRecv error: recv got EOF from 127.0.98.1:54909 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260927 02:53:11.351907 9224 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:53:11.415810 9228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:11.416006 9228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:11.416026 9228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:11.417088 9228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:11.417117 9228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:11.417140 9228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:11.418452 9228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43629
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:11.418613 9228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:11.418788 9228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:11.423403 9233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:11.423389 9236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:11.423390 9234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:11.423692 9228 server_base.cc:1034] running on GCE node
I20260927 02:53:11.423923 9228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:11.424131 9228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:11.425285 9228 hybrid_clock.cc:648] HybridClock initialized: now 1790477591425266 us; error 32 us; skew 500 ppm
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:11.427570 9228 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:53:11.428002 9228 webserver.cc:460] Webserver started at http://127.0.98.2:43629/ using document root <none> and password file <none>
I20260927 02:53:11.428182 9228 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:11.428239 9228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:11.429431 9228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:11.429987 9243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.430143 9228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.430219 9228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "ae20c48e53d145099fa5d18b5c0a3ab1"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "72dbc9eb73b6c56e36432bc7076bb9e6"
server_key_iv: "cc5261cc40c7bc89b6282b3c0bf56f5d"
server_key_version: "encryptionkey@0"
I20260927 02:53:11.430552 9228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:11.445883 9228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:11.447997 9228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:11.448129 9228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:11.448370 9228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:11.448691 9228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:11.448724 9228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.448751 9228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:11.448774 9228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.454118 9228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41993
I20260927 02:53:11.454164 9356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41993 every 8 connection(s)
I20260927 02:53:11.454476 9228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.460815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 9228
I20260927 02:53:11.461119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43329
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36627
--webserver_port=35115
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:11.461349 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.455127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54283 (local address 127.0.98.62:36627)
0927 02:53:11.455171 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.455174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.455379 (+ 205us) server_negotiation.cc:407] Connection header received
0927 02:53:11.455850 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.455851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.455862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.455905 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.456721 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.457195 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.458179 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.458362 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.458408 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.461288 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54283: BlockingRecv error: recv got EOF from 127.0.98.2:54283 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:53:11.461386 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.455121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:54283)
0927 02:53:11.455270 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.455303 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.455742 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.455965 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.455969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.456474 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.456657 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.456661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.457273 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.457274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.458105 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.458107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.458436 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.458437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.458540 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.459027 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.459035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.461188 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
W20260927 02:53:11.461735 9357 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36627 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36627: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.466324 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.461977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:55691)
0927 02:53:11.462050 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.462059 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.462096 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.462198 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.462199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.462313 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.462358 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.462359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.462967 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.462968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.463679 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.463681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.463996 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.463997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.464059 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.464440 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.464445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.466258 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 02:53:11.466406 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.462058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55691 (local address 127.0.98.62:36627)
0927 02:53:11.462111 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.462112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.462120 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:11.462142 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.462143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.462152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.462195 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.462419 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.462912 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.463750 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.463933 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.463989 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.466348 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55691: BlockingRecv error: recv got EOF from 127.0.98.2:55691 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.470785 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.466556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:40197)
0927 02:53:11.466600 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.466609 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.466642 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.466774 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.466775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.466898 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.466929 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.466930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.467513 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.467514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.468242 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.468244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.468589 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.468591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.468640 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.469000 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.469005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.470697 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:53:11.470876 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.466614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40197 (local address 127.0.98.62:36627)
0927 02:53:11.466648 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.466650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.466659 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:11.466681 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.466682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.466690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.466741 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.467003 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.467443 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.468329 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.468521 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.468580 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.470825 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40197: BlockingRecv error: recv got EOF from 127.0.98.2:40197 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 02:53:11.470929 9357 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:53:11.540704 9360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:11.540898 9360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:11.540927 9360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:11.542045 9360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:11.542083 9360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:11.542116 9360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:11.543490 9360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35115
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:11.543708 9360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:11.543898 9360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:11.548689 9366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:11.548708 9365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:11.548718 9368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:11.548908 9360 server_base.cc:1034] running on GCE node
I20260927 02:53:11.549094 9360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:11.549294 9360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:11.550443 9360 hybrid_clock.cc:648] HybridClock initialized: now 1790477591550430 us; error 33 us; skew 500 ppm
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:11.552779 9360 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:53:11.553162 9360 webserver.cc:460] Webserver started at http://127.0.98.3:35115/ using document root <none> and password file <none>
I20260927 02:53:11.553382 9360 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:11.553439 9360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:11.554695 9360 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:11.555197 9375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.555330 9360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.555416 9360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "96e1d0db893b413cac8423bee1c2fdc5"
format_stamp: "Formatted at 2026-09-27 02:53:10 on dist-test-slave-mfx9"
server_key: "c82e3e45170c84e3e68ac09661a4b7ab"
server_key_iv: "e324bbaeaacca2c5c4c8b78316b6275b"
server_key_version: "encryptionkey@0"
I20260927 02:53:11.555754 9360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:11.570406 9360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:11.572417 9360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:11.572538 9360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:11.572728 9360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:11.573031 9360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:11.573062 9360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.573082 9360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:11.573104 9360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:11.578214 9360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43329
I20260927 02:53:11.578274 9488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43329 every 8 connection(s)
I20260927 02:53:11.578570 9360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.585000 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.579227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:48933)
0927 02:53:11.579366 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.579400 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.579844 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.580047 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.580050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.580591 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.580780 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.580784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.581368 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.581369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.582146 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.582148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.582463 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.582464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.582555 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.583001 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.583009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.584850 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
W20260927 02:53:11.585000 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.579222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48933 (local address 127.0.98.62:36627)
0927 02:53:11.579273 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.579275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.579475 (+ 200us) server_negotiation.cc:407] Connection header received
0927 02:53:11.579919 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.579920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.579929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.579975 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.580843 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.581312 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.582224 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.582406 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.582457 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.584954 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48933: BlockingRecv error: recv got EOF from 127.0.98.3:48933 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 02:53:11.585263 9489 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36627 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36627: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 02:53:11.585989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 9360
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.589917 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.585469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:57887)
0927 02:53:11.585509 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.585519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.585551 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.585725 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.585726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.585822 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.585863 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.585864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.586663 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.586663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.587414 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.587417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.587816 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.587816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.587858 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.588198 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.588203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.589856 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:53:11.589996 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.585598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57887 (local address 127.0.98.62:36627)
0927 02:53:11.585639 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.585641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.585646 (+ 5us) server_negotiation.cc:407] Connection header received
0927 02:53:11.585666 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.585667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.585674 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.585717 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.585924 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.586583 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.587472 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.587751 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.587810 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.589933 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57887: BlockingRecv error: recv got EOF from 127.0.98.3:57887 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:53:11.593180 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.586342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.0.98.62:36627)
0927 02:53:11.586443 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.586445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.586544 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:53:11.586583 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.586583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.586593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.586651 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.587306 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.587944 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.588908 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.589170 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.589235 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.590344 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:11.590349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:11.590349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:11.590361 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:11.591826 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:11.592175 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:11.592178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:11.592179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:11.592301 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:11.592497 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:11.592498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:11.592499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:11.592792 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:11.592934 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:11.593023 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:53:11.593103 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:11 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:11.594224 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:11.590348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:43993)
0927 02:53:11.590405 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:11.590415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:11.590450 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:11.590552 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:11.590553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:11.590661 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:11.590683 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.590684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.591270 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.591271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.591743 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:11.591745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.592119 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.592120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:11.592172 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.592505 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:11.592509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:11.594167 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:53:11.594276 9227 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:11.590399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43993 (local address 127.0.98.62:36627)
0927 02:53:11.590434 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:11.590436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:11.590442 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:53:11.590500 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:11.590501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:11.590509 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:11.590551 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:11.590743 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.591208 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.591806 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:11.592044 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:11.592105 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:11.594233 (+ 2128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43993: BlockingRecv error: recv got EOF from 127.0.98.3:43993 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:53:11.594378 9489 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:53:12 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:12 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:12.362871 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:12.352282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:47441)
0927 02:53:12.352434 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:12.352444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:12.352496 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:12.352627 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:12.352628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:12.352778 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:12.352826 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:12.352828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.353488 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.353489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:12.354030 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:12.354032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.354410 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.354411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:12.354451 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:12.361171 (+ 6720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:12.361176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:12.362802 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
W20260927 02:53:12.362995 9495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:12.352362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47441 (local address 127.0.98.62:36627)
0927 02:53:12.352471 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:12.352473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:12.352482 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:12.352550 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:12.352550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:12.352560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:12.352612 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:12.352910 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.353416 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.354125 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.354321 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.354410 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:12.362918 (+ 8508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47441: BlockingRecv error: recv got EOF from 127.0.98.1:47441 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
Sep 27 02:53:12 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:12 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:12.475625 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:12.471277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:42273)
0927 02:53:12.471385 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:12.471395 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:12.471449 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:12.471560 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:12.471561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:12.471709 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:12.471756 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:12.471757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.472383 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.472384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:12.472963 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:12.472965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.473303 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.473304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:12.473339 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:12.473815 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:12.473819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:12.475539 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
W20260927 02:53:12.475678 9495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:12.471346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42273 (local address 127.0.98.62:36627)
0927 02:53:12.471406 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:12.471408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:12.471440 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:53:12.471489 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:12.471489 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:12.471499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:12.471548 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:12.471806 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.472315 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.473055 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.473244 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.473306 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:12.475627 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42273: BlockingRecv error: recv got EOF from 127.0.98.2:42273 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 02:53:12 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:12 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:12.599038 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:12.594802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:41823)
0927 02:53:12.594907 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:12.594918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:12.594968 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:12.595103 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:12.595104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:12.595289 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:12.595357 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:12.595360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.595975 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.595975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:12.596508 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:12.596509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.596835 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.596836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:12.596876 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:12.597270 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:12.597274 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:12.598970 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
W20260927 02:53:12.599118 9495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:12.594834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41823 (local address 127.0.98.62:36627)
0927 02:53:12.594883 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:12.594885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:12.594969 (+ 84us) server_negotiation.cc:407] Connection header received
0927 02:53:12.595016 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:12.595016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:12.595032 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:12.595087 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:12.595435 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.595910 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.596576 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:12.596782 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:12.596832 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:12.599057 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41823: BlockingRecv error: recv got EOF from 127.0.98.3:41823 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 02:53:13 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:13 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:13.367695 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:13.363344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:33327)
0927 02:53:13.363478 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:13.363489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:13.363542 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:13.363673 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:13.363674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:13.363854 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:13.363910 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:13.363912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.364516 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.364516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:13.365062 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:13.365064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.365366 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.365367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:13.365415 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:13.365864 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:13.365868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:13.367613 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
W20260927 02:53:13.367785 9499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:13.363405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33327 (local address 127.0.98.62:36627)
0927 02:53:13.363522 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:13.363524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:13.363532 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:13.363590 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:13.363590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:13.363600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:13.363659 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:13.363970 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.364452 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.365128 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.365301 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.365341 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:13.367712 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33327: BlockingRecv error: recv got EOF from 127.0.98.1:33327 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
Sep 27 02:53:13 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:13 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:13.480291 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:13.476124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:39239)
0927 02:53:13.476247 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:13.476257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:13.476307 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:13.476416 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:13.476416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:13.476565 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:13.476612 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:13.476613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.477213 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.477213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:13.477720 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:13.477721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.478022 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.478023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:13.478057 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:13.478476 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:13.478480 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:13.480206 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
W20260927 02:53:13.480358 9499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:13.476197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39239 (local address 127.0.98.62:36627)
0927 02:53:13.476268 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:13.476269 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:13.476309 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:53:13.476360 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:13.476361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:13.476371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:13.476410 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:13.476659 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.477155 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.477777 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.477967 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.478016 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:13.480298 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39239: BlockingRecv error: recv got EOF from 127.0.98.2:39239 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 02:53:13 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:13 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:13.603965 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:13.599450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:48849)
0927 02:53:13.599570 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:13.599580 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:13.599635 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:13.599887 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:13.599888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:13.600062 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:13.600110 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:13.600112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.600768 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.600768 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:13.601271 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:13.601272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.601659 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.601660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:13.601717 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:13.602155 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:13.602159 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:13.603908 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
W20260927 02:53:13.604055 9499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:13.599623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48849 (local address 127.0.98.62:36627)
0927 02:53:13.599721 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:13.599723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:13.599731 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:13.599752 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:13.599753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:13.599765 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:13.599871 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:13.600194 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.600709 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.601341 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:13.601571 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:13.601637 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:13.603992 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48849: BlockingRecv error: recv got EOF from 127.0.98.3:48849 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 02:53:14 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:14 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:14.372577 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:14.368180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:48725)
0927 02:53:14.368300 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:14.368310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:14.368357 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:14.368501 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:14.368502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:14.368657 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:14.368706 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:14.368708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.369438 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.369439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:14.369959 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:14.369961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.370295 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.370296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:14.370355 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:14.370866 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:14.370871 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:14.372511 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
W20260927 02:53:14.372665 9503 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:14.368261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48725 (local address 127.0.98.62:36627)
0927 02:53:14.368378 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:14.368380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:14.368388 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:14.368414 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:14.368415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:14.368426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:14.368492 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:14.368799 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.369362 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.370037 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.370226 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.370285 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:14.372597 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48725: BlockingRecv error: recv got EOF from 127.0.98.1:48725 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
Sep 27 02:53:14 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:14 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:14.484990 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:14.480787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:48403)
0927 02:53:14.480917 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:14.480927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:14.480976 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:14.481112 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:14.481113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:14.481255 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:14.481302 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:14.481303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.481895 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.481896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:14.482419 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:14.482421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.482780 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.482781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:14.482824 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:14.483252 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:14.483255 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:14.484935 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
W20260927 02:53:14.485096 9503 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:14.480903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48403 (local address 127.0.98.62:36627)
0927 02:53:14.480951 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:14.480952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:14.480964 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:53:14.481042 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:14.481042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:14.481054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:14.481105 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:14.481354 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.481841 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.482499 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.482692 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.482775 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:14.485031 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48403: BlockingRecv error: recv got EOF from 127.0.98.2:48403 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 02:53:14 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:14 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:14.608855 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:14.604398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:39907)
0927 02:53:14.604540 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:14.604550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:14.604609 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:14.604737 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:14.604737 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:14.604894 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:14.604985 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:14.604987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.605602 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.605603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:14.606208 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:14.606211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.606634 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.606635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:14.606691 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:14.607118 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:14.607122 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:14.608784 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
W20260927 02:53:14.608914 9503 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:14.604478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39907 (local address 127.0.98.62:36627)
0927 02:53:14.604544 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:14.604546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:14.604605 (+ 59us) server_negotiation.cc:407] Connection header received
0927 02:53:14.604669 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:14.604670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:14.604683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:14.604724 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:14.605043 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.605528 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.606292 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:14.606563 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:14.606625 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:14.608865 (+ 2240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39907: BlockingRecv error: recv got EOF from 127.0.98.3:39907 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 02:53:15 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:15 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:15.377123 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:15.372987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:43017)
0927 02:53:15.373118 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:15.373130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:15.373190 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:15.373311 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:15.373312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:15.373482 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:15.373529 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:15.373531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.374150 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.374150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:15.374673 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:15.374675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.374992 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.374993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:15.375036 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:15.375485 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:15.375490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:15.377065 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
W20260927 02:53:15.377216 9507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:15.373035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43017 (local address 127.0.98.62:36627)
0927 02:53:15.373144 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:15.373145 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:15.373175 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:53:15.373229 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:15.373230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:15.373239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:15.373308 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:15.373583 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.374093 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.374735 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.374934 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.374983 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:15.377149 (+ 2166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43017: BlockingRecv error: recv got EOF from 127.0.98.1:43017 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
Sep 27 02:53:15 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:15 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:15.489724 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:15.485430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:44977)
0927 02:53:15.485571 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:15.485582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:15.485649 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:15.485765 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:15.485766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:15.485914 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:15.485993 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:15.485995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.486605 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.486606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:15.487198 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:15.487201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.487532 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.487534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:15.487579 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:15.488005 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:15.488009 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:15.489639 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
W20260927 02:53:15.489791 9507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:15.485501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44977 (local address 127.0.98.62:36627)
0927 02:53:15.485563 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:15.485565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:15.485646 (+ 81us) server_negotiation.cc:407] Connection header received
0927 02:53:15.485701 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:15.485702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:15.485713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:15.485749 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:15.486051 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.486542 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.487272 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.487468 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.487524 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:15.489739 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44977: BlockingRecv error: recv got EOF from 127.0.98.2:44977 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 02:53:15 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:15 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:15.613663 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:15.609334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:51927)
0927 02:53:15.609485 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:15.609496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:15.609576 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:15.609712 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:15.609712 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:15.609881 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:15.609926 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:15.609928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.610559 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.610560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:15.611221 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:15.611223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.611580 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.611580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:15.611633 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:15.612034 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:15.612039 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:15.613606 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
W20260927 02:53:15.613750 9507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:15.609392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51927 (local address 127.0.98.62:36627)
0927 02:53:15.609464 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:15.609466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:15.609569 (+ 103us) server_negotiation.cc:407] Connection header received
0927 02:53:15.609622 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:15.609623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:15.609636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:15.609687 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:15.610019 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.610506 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.611297 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:15.611510 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:15.611575 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:15.613687 (+ 2112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51927: BlockingRecv error: recv got EOF from 127.0.98.3:51927 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 02:53:16 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:16 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:16.381783 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:16.377558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:46313)
0927 02:53:16.377672 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:16.377682 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:16.377732 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:16.377864 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:16.377864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:16.378016 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:16.378065 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:16.378066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.378672 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.378673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:16.379220 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:16.379222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.379592 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.379594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:16.379653 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:16.380128 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:16.380132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:16.381725 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
W20260927 02:53:16.381875 9511 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:16.377625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46313 (local address 127.0.98.62:36627)
0927 02:53:16.377753 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:16.377754 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:16.377762 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:16.377783 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:16.377784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:16.377793 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:16.377848 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:16.378116 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.378613 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.379281 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.379501 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.379583 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:16.381809 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46313: BlockingRecv error: recv got EOF from 127.0.98.1:46313 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
Sep 27 02:53:16 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:16 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:16.494326 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:16.490195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:38265)
0927 02:53:16.490315 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:16.490327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:16.490388 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:16.490522 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:16.490523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:16.490680 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:16.490725 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:16.490727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.491345 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.491346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:16.491875 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:16.491877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.492187 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.492188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:16.492238 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:16.492661 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:16.492665 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:16.494237 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
W20260927 02:53:16.494391 9511 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:16.490307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38265 (local address 127.0.98.62:36627)
0927 02:53:16.490360 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:16.490362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:16.490371 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:16.490448 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:16.490448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:16.490460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:16.490517 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:16.490800 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.491281 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.491950 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.492131 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.492184 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:16.494340 (+ 2156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38265: BlockingRecv error: recv got EOF from 127.0.98.2:38265 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 02:53:16 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:16 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:16.618201 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:16.614051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:55805)
0927 02:53:16.614207 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:16.614218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:16.614284 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:16.614419 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:16.614420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:16.614583 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:16.614632 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:16.614634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.615230 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.615231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:16.615793 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:16.615796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.616077 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.616078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:16.616118 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:16.616503 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:16.616507 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:16.618127 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
W20260927 02:53:16.618266 9511 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:16.614111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55805 (local address 127.0.98.62:36627)
0927 02:53:16.614172 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:16.614175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:16.614275 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:53:16.614344 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:16.614345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:16.614356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:16.614400 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:16.614686 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.615174 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.615860 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:16.616028 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:16.616067 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:16.618215 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55805: BlockingRecv error: recv got EOF from 127.0.98.3:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 02:53:17 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:17 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:17.392350 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:17.382229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:55661)
0927 02:53:17.382368 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:17.382381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:17.382460 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:17.382612 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:17.382613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:17.382801 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:17.382862 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:17.382864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.383465 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.383465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:17.384139 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:17.384142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.384474 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.384475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:17.384515 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:17.390594 (+ 6079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:17.390598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:17.392281 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
W20260927 02:53:17.392491 9515 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:17.382314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55661 (local address 127.0.98.62:36627)
0927 02:53:17.382438 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:17.382440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:17.382448 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:17.382529 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:17.382529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:17.382540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:17.382586 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:17.382918 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.383402 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.384216 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.384419 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.384468 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:17.392408 (+ 7940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55661: BlockingRecv error: recv got EOF from 127.0.98.1:55661 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
Sep 27 02:53:17 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:17 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:17.499154 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:17.494824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:36839)
0927 02:53:17.494969 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:17.494983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:17.495052 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:17.495181 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:17.495182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:17.495350 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:17.495400 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:17.495402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.495998 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.495999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:17.496530 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:17.496532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.496892 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.496893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:17.496945 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:17.497375 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:17.497379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:17.499074 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
W20260927 02:53:17.499219 9515 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:17.494852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36839 (local address 127.0.98.62:36627)
0927 02:53:17.494895 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:17.494897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:17.495043 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:53:17.495108 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:17.495109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:17.495119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:17.495167 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:17.495450 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.495925 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.496605 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.496808 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.496854 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:17.499168 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36839: BlockingRecv error: recv got EOF from 127.0.98.2:36839 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 02:53:17 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:17 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:17.622890 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:17.618687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:53597)
0927 02:53:17.618823 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:17.618835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:17.618899 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:17.619028 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:17.619029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:17.619209 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:17.619265 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:17.619267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.619899 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.619900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:17.620420 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:17.620422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.620716 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.620716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:17.620755 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:17.621211 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:17.621215 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:17.622823 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
W20260927 02:53:17.623008 9515 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:17.618714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53597 (local address 127.0.98.62:36627)
0927 02:53:17.618793 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:17.618795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:17.618884 (+ 89us) server_negotiation.cc:407] Connection header received
0927 02:53:17.618950 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:17.618951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:17.618966 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:17.619020 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:17.619338 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.619837 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.620481 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:17.620662 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:17.620708 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:17.622944 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53597: BlockingRecv error: recv got EOF from 127.0.98.3:53597 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 02:53:18 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:18 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:18.396899 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:18.392790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:38781)
0927 02:53:18.392934 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:18.392956 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:18.393017 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:18.393136 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:18.393137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:18.393289 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:18.393337 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:18.393338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.393981 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.393982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:18.394520 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:18.394521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.394839 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.394840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:18.394884 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:18.395316 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:18.395320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:18.396842 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
W20260927 02:53:18.396993 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:18.392863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38781 (local address 127.0.98.62:36627)
0927 02:53:18.392997 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:18.392999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:18.393006 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:53:18.393060 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:18.393060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:18.393070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:18.393125 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:18.393387 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.393897 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.394582 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.394785 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.394830 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:18.396924 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38781: BlockingRecv error: recv got EOF from 127.0.98.1:38781 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Sep 27 02:53:18 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:18 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:18.503647 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:18.499604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:45171)
0927 02:53:18.499740 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:18.499752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:18.499812 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:18.499918 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:18.499919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:18.500080 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:18.500127 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:18.500128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.500720 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.500721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:18.501262 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:18.501263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.501550 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.501550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:18.501586 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:18.502002 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:18.502006 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:18.503591 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
W20260927 02:53:18.503734 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:18.499665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45171 (local address 127.0.98.62:36627)
0927 02:53:18.499731 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:18.499733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:18.499804 (+ 71us) server_negotiation.cc:407] Connection header received
0927 02:53:18.499856 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:18.499856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:18.499867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:18.499912 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:18.500174 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.500664 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.501319 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.501497 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.501546 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:18.503676 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45171: BlockingRecv error: recv got EOF from 127.0.98.2:45171 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 02:53:18 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:18 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:18.627840 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:18.623431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:56921)
0927 02:53:18.623581 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:18.623594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:18.623673 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:18.623793 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:18.623794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:18.623961 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:18.624005 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:18.624006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.624651 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.624653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:18.625217 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:18.625219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.625570 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.625571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:18.625631 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:18.626116 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:18.626121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:18.627784 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260927 02:53:18.627926 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:18.623549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56921 (local address 127.0.98.62:36627)
0927 02:53:18.623603 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:18.623605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:18.623652 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:53:18.623724 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:18.623725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:18.623738 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:18.623783 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:18.624077 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.624571 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.625288 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:18.625485 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:18.625551 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:18.627862 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56921: BlockingRecv error: recv got EOF from 127.0.98.3:56921 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 02:53:19 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:19 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:19.401556 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:19.397326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:45069)
0927 02:53:19.397471 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:19.397481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:19.397542 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:19.397697 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:19.397698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:19.397887 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:19.397936 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:19.397938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.398561 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.398562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:19.399104 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:19.399106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.399380 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.399380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:19.399416 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:19.399853 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:19.399858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:19.401472 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
W20260927 02:53:19.401628 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:19.397442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45069 (local address 127.0.98.62:36627)
0927 02:53:19.397555 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:19.397556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:19.397583 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:53:19.397604 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:19.397604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:19.397630 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:19.397677 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:19.397993 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.398490 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.399156 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.399334 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.399375 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:19.401571 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45069: BlockingRecv error: recv got EOF from 127.0.98.1:45069 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
Sep 27 02:53:19 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:19 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:19.508631 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:19.504073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:49669)
0927 02:53:19.504219 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:19.504231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:19.504299 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:19.504424 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:19.504425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:19.504594 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:19.504639 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:19.504640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.505413 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.505414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:19.505933 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:19.505934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.506252 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.506253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:19.506304 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:19.506818 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:19.506822 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:19.508575 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
W20260927 02:53:19.508711 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:19.504155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49669 (local address 127.0.98.62:36627)
0927 02:53:19.504218 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:19.504220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:19.504280 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:53:19.504356 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:19.504357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:19.504368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:19.504415 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:19.504709 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.505337 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.506005 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.506187 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.506240 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:19.508652 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49669: BlockingRecv error: recv got EOF from 127.0.98.2:49669 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 02:53:19 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:19 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:19.632308 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:19.628243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:43663)
0927 02:53:19.628367 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:19.628377 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:19.628434 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:19.628557 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:19.628558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:19.628702 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:19.628748 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:19.628749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.629349 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.629350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:19.629886 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:19.629888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.630180 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.630180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:19.630218 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:19.630679 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:19.630683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:19.632251 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
W20260927 02:53:19.632392 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:19.628291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43663 (local address 127.0.98.62:36627)
0927 02:53:19.628338 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:19.628340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:19.628421 (+ 81us) server_negotiation.cc:407] Connection header received
0927 02:53:19.628496 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:19.628496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:19.628507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:19.628554 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:19.628795 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.629283 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.629941 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:19.630131 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:19.630176 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:19.632332 (+ 2156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43663: BlockingRecv error: recv got EOF from 127.0.98.3:43663 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 02:53:20 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:20 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:20.406247 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:20.401989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:35943)
0927 02:53:20.402114 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:20.402125 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:20.402172 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:20.402315 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:20.402316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:20.402469 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:20.402518 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:20.402520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.403155 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.403155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:20.403664 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:20.403666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.403940 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.403940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:20.403979 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:20.404463 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:20.404467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:20.406188 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
W20260927 02:53:20.406343 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:20.402070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35943 (local address 127.0.98.62:36627)
0927 02:53:20.402194 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:20.402196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:20.402204 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:20.402228 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:20.402228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:20.402238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:20.402302 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:20.402574 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.403101 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.403719 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.403896 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.403938 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:20.406272 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35943: BlockingRecv error: recv got EOF from 127.0.98.1:35943 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
Sep 27 02:53:20 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:20 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:20.513466 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:20.509034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:45023)
0927 02:53:20.509145 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:20.509156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:20.509223 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:20.509388 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:20.509388 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:20.509552 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:20.509602 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:20.509603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.510228 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.510228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:20.510810 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:20.510812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.511131 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.511132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:20.511170 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:20.511661 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:20.511667 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:20.513390 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
W20260927 02:53:20.513541 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:20.509095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45023 (local address 127.0.98.62:36627)
0927 02:53:20.509163 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:20.509165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:20.509215 (+ 50us) server_negotiation.cc:407] Connection header received
0927 02:53:20.509290 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:20.509291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:20.509304 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:20.509365 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:20.509655 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.510161 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.510881 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.511077 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.511125 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:20.513481 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45023: BlockingRecv error: recv got EOF from 127.0.98.2:45023 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 02:53:20 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:20 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:20.636951 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:20.632721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:36557)
0927 02:53:20.632868 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:20.632880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:20.632954 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:20.633105 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:20.633106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:20.633280 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:20.633333 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:20.633335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.633979 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.633980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:20.634504 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:20.634506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.634837 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.634838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:20.634896 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:20.635272 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:20.635276 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:20.636893 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
W20260927 02:53:20.637012 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:20.632809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36557 (local address 127.0.98.62:36627)
0927 02:53:20.632900 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:20.632902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:20.632934 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:53:20.633027 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:20.633028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:20.633041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:20.633091 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:20.633405 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.633907 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.634575 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:20.634776 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:20.634837 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:20.636965 (+ 2128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36557: BlockingRecv error: recv got EOF from 127.0.98.3:36557 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 02:53:21 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:21 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:21.411376 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:21.406782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:46813)
0927 02:53:21.406938 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:21.406950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:21.407009 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:21.407158 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:21.407159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:21.407347 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:21.407393 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:21.407395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.408074 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.408075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:21.408679 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:21.408681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.409040 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.409041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:21.409097 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:21.409531 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:21.409535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:21.411319 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
W20260927 02:53:21.411489 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:21.406802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46813 (local address 127.0.98.62:36627)
0927 02:53:21.406930 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:21.406932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:21.406996 (+ 64us) server_negotiation.cc:407] Connection header received
0927 02:53:21.407068 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:21.407068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:21.407080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:21.407139 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:21.407467 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.407993 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.408777 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.408969 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.409022 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:21.411403 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46813: BlockingRecv error: recv got EOF from 127.0.98.1:46813 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
Sep 27 02:53:21 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:21 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:21.517995 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:21.513889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:35657)
0927 02:53:21.513993 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:21.514003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:21.514065 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:21.514189 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:21.514189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:21.514344 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:21.514391 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:21.514393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.515037 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.515038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:21.515580 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:21.515582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.515885 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.515886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:21.515924 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:21.516340 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:21.516345 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:21.517938 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":54,"threads_started":1}
W20260927 02:53:21.518100 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:21.513934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35657 (local address 127.0.98.62:36627)
0927 02:53:21.514008 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:21.514009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:21.514057 (+ 48us) server_negotiation.cc:407] Connection header received
0927 02:53:21.514115 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:21.514116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:21.514126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:21.514175 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:21.514438 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.514967 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.515645 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.515829 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.515881 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:21.518038 (+ 2157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35657: BlockingRecv error: recv got EOF from 127.0.98.2:35657 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 02:53:21 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:21 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:21.641647 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:21.637372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:51743)
0927 02:53:21.637513 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:21.637524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:21.637577 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:21.637693 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:21.637694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:21.637841 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:21.637887 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:21.637889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.638503 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.638504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:21.639063 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:21.639065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.639346 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.639346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:21.639379 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:21.639789 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:21.639793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:21.641587 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":58,"threads_started":1}
W20260927 02:53:21.641697 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:21.637477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51743 (local address 127.0.98.62:36627)
0927 02:53:21.637528 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:21.637529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:21.637565 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:53:21.637623 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:21.637624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:21.637634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:21.637682 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:21.637943 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.638431 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.639120 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:21.639296 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:21.639342 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:21.641652 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51743: BlockingRecv error: recv got EOF from 127.0.98.3:51743 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 02:53:22 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:22 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:22.419524 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:22.411824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:38869)
0927 02:53:22.411971 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:22.411983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:22.412036 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:22.412215 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:22.412216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:22.412390 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:22.412438 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:22.412439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.413068 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.413069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:22.413657 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:22.413659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.414002 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.414003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:22.414055 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:22.417770 (+ 3715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:22.417776 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:22.419423 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
W20260927 02:53:22.419588 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:22.411897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38869 (local address 127.0.98.62:36627)
0927 02:53:22.412011 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:22.412013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:22.412024 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:53:22.412106 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:22.412107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:22.412117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:22.412170 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:22.412503 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.412992 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.413739 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.413930 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.413989 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:22.419528 (+ 5539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38869: BlockingRecv error: recv got EOF from 127.0.98.1:38869 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
Sep 27 02:53:22 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:22 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:22.522702 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:22.518475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:52841)
0927 02:53:22.518615 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:22.518624 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:22.518677 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:22.518824 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:22.518824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:22.519000 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:22.519050 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:22.519052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.519653 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.519654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:22.520199 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:22.520201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.520513 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.520514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:22.520565 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:22.521055 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:22.521059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:22.522640 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
W20260927 02:53:22.522809 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:22.518524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52841 (local address 127.0.98.62:36627)
0927 02:53:22.518597 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:22.518599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:22.518695 (+ 96us) server_negotiation.cc:407] Connection header received
0927 02:53:22.518722 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:22.518722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:22.518733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:22.518816 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:22.519104 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.519590 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.520258 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.520436 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.520480 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:22.522730 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52841: BlockingRecv error: recv got EOF from 127.0.98.2:52841 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 02:53:22 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:22 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:22.646859 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:22.642098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:53085)
0927 02:53:22.642220 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:22.642230 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:22.642293 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:22.642407 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:22.642408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:22.642587 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:22.642629 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:22.642631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.643308 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.643309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:22.643841 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:22.643843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.644166 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.644166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:22.644220 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:22.644609 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:22.644613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:22.646691 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260927 02:53:22.646970 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:22.642161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53085 (local address 127.0.98.62:36627)
0927 02:53:22.642214 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:22.642216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:22.642277 (+ 61us) server_negotiation.cc:407] Connection header received
0927 02:53:22.642347 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:22.642348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:22.642360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:22.642402 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:22.642702 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.643236 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.643914 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:22.644108 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:22.644162 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:22.646903 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53085: BlockingRecv error: recv got EOF from 127.0.98.3:53085 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 02:53:23 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:23 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:23.424167 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:23.420018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:45269)
0927 02:53:23.420130 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:23.420143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:23.420211 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:23.420328 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:23.420329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:23.420492 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:23.420547 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:23.420549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.421187 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.421188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:23.421722 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:23.421723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.422019 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.422019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:23.422055 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:23.422447 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:23.422451 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:23.424080 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
W20260927 02:53:23.424243 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:23.420089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45269 (local address 127.0.98.62:36627)
0927 02:53:23.420218 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:23.420220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:23.420229 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:23.420250 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:23.420251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:23.420260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:23.420315 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:23.420614 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.421122 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.421784 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.421970 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.422015 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:23.424174 (+ 2159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45269: BlockingRecv error: recv got EOF from 127.0.98.1:45269 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
Sep 27 02:53:23 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:23 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:23.528429 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:23.523180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:48601)
0927 02:53:23.523316 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:23.523327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:23.523413 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:23.523558 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:23.523559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:23.523773 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:23.523821 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:23.523823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.524844 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.524845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:23.525526 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:23.525529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.525988 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.525988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:23.526039 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:23.526504 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:23.526509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:23.528363 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
W20260927 02:53:23.528532 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:23.523256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48601 (local address 127.0.98.62:36627)
0927 02:53:23.523304 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:23.523306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:23.523406 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:53:23.523481 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:23.523482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:23.523495 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:23.523545 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:23.523963 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.524748 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.525602 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.525917 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.525991 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:23.528467 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48601: BlockingRecv error: recv got EOF from 127.0.98.2:48601 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 02:53:23 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:23 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:23.651937 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:23.647400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:59595)
0927 02:53:23.647526 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:23.647536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:23.647585 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:23.647713 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:23.647714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:23.647881 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:23.647934 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:23.647936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.648530 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.648531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:23.649068 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:23.649070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.649418 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.649418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:23.649454 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:23.649926 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:23.649932 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:23.651860 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
W20260927 02:53:23.652009 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:23.647505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59595 (local address 127.0.98.62:36627)
0927 02:53:23.647566 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:23.647568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:23.647577 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:23.647642 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:23.647643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:23.647654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:23.647702 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:23.647990 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.648471 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.649172 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:23.649354 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:23.649411 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:23.651947 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59595: BlockingRecv error: recv got EOF from 127.0.98.3:59595 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 02:53:24 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:24 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:24.428737 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:24.424648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:48623)
0927 02:53:24.424787 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:24.424796 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:24.424843 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:24.424967 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:24.424967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:24.425116 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:24.425163 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:24.425164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.425788 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.425788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:24.426314 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:24.426316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.426640 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.426641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:24.426677 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:24.427135 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:24.427139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:24.428679 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
W20260927 02:53:24.428839 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:24.424712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48623 (local address 127.0.98.62:36627)
0927 02:53:24.424828 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:24.424829 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:24.424838 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:24.424885 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:24.424885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:24.424894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:24.424954 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:24.425215 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.425731 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.426383 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.426589 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.426639 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:24.428764 (+ 2125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48623: BlockingRecv error: recv got EOF from 127.0.98.1:48623 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
Sep 27 02:53:24 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:24 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:24.533018 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:24.528966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:33531)
0927 02:53:24.529098 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:24.529110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:24.529172 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:24.529283 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:24.529284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:24.529432 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:24.529480 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:24.529482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.530074 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.530075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:24.530622 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:24.530627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.530934 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.530934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:24.530975 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:24.531381 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:24.531385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:24.532963 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
W20260927 02:53:24.533128 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:24.529080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33531 (local address 127.0.98.62:36627)
0927 02:53:24.529124 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:24.529126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:24.529156 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:53:24.529223 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:24.529224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:24.529234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:24.529269 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:24.529549 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.530022 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.530683 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.530881 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.530928 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:24.533060 (+ 2132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33531: BlockingRecv error: recv got EOF from 127.0.98.2:33531 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 02:53:24 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:24 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:24.656708 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:24.652388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:56641)
0927 02:53:24.652516 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:24.652527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:24.652585 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:24.652703 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:24.652704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:24.652862 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:24.652909 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:24.652911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.653484 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.653484 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:24.654016 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:24.654018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.654283 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.654283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:24.654318 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:24.654722 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:24.654726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:24.656638 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
W20260927 02:53:24.656791 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:24.652430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56641 (local address 127.0.98.62:36627)
0927 02:53:24.652485 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:24.652487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:24.652578 (+ 91us) server_negotiation.cc:407] Connection header received
0927 02:53:24.652640 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:24.652640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:24.652652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:24.652689 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:24.652959 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.653427 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.654074 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:24.654238 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:24.654278 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:24.656731 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56641: BlockingRecv error: recv got EOF from 127.0.98.3:56641 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 02:53:25 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:25 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:25.433336 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:25.429171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:37633)
0927 02:53:25.429290 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:25.429301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:25.429355 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:25.429481 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:25.429482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:25.429646 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:25.429693 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:25.429695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.430294 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.430295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:25.430924 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:25.430926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.431269 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.431270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:25.431320 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:25.431712 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:25.431716 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:25.433260 (+ 1544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
W20260927 02:53:25.433403 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:25.429262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37633 (local address 127.0.98.62:36627)
0927 02:53:25.429375 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:25.429376 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:25.429384 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:25.429405 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:25.429406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:25.429415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:25.429471 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:25.429748 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.430241 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.431000 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.431210 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.431263 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:25.433346 (+ 2083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37633: BlockingRecv error: recv got EOF from 127.0.98.1:37633 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":49,"threads_started":1}
Sep 27 02:53:25 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:25 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:25.537760 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:25.533440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:42533)
0927 02:53:25.533560 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:25.533572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:25.533633 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:25.533740 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:25.533741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:25.533889 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:25.533933 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:25.533935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.534591 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.534593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:25.535214 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:25.535216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.535551 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.535552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:25.535602 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:25.536022 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:25.536026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:25.537664 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
W20260927 02:53:25.537822 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:25.533500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42533 (local address 127.0.98.62:36627)
0927 02:53:25.533564 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:25.533566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:25.533619 (+ 53us) server_negotiation.cc:407] Connection header received
0927 02:53:25.533678 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:25.533678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:25.533689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:25.533736 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:25.534000 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.534508 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.535289 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.535481 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.535535 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:25.537761 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42533: BlockingRecv error: recv got EOF from 127.0.98.2:42533 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 02:53:25 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:25 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:25.661386 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:25.657109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:35427)
0927 02:53:25.657219 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:25.657231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:25.657284 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:25.657408 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:25.657409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:25.657568 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:25.657611 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:25.657613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.658306 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.658307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:25.658871 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:25.658873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.659259 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.659260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:25.659299 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:25.659751 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:25.659755 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:25.661311 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
W20260927 02:53:25.661458 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:25.657205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35427 (local address 127.0.98.62:36627)
0927 02:53:25.657259 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:25.657260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:25.657269 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:25.657337 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:25.657338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:25.657352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:25.657396 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:25.657674 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.658226 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.658959 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:25.659198 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:25.659259 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:25.661405 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35427: BlockingRecv error: recv got EOF from 127.0.98.3:35427 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 02:53:26 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:26 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:26.437845 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:26.433792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:40687)
0927 02:53:26.433909 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:26.433919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:26.433968 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:26.434085 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:26.434086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:26.434234 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:26.434280 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:26.434282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.434900 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.434901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:26.435423 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:26.435425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.435724 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.435724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:26.435761 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.436153 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:26.436156 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:26.437757 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20260927 02:53:26.437907 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:26.433878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40687 (local address 127.0.98.62:36627)
0927 02:53:26.433978 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:26.433979 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:26.433988 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:26.434010 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:26.434011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:26.434020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:26.434081 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:26.434331 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.434844 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.435492 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.435674 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.435719 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.437849 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40687: BlockingRecv error: recv got EOF from 127.0.98.1:40687 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":42,"threads_started":1}
Sep 27 02:53:26 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:53:26 dist-test-slave-mfx9 krb5kdc[8493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:53:26.542498 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:26.538243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:38673)
0927 02:53:26.538353 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:26.538365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:26.538431 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:26.538576 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:26.538577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:26.538741 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:26.538811 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:26.538812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.539452 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.539454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:26.539977 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:26.539979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.540314 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.540315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:26.540365 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.540784 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:26.540787 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:26.542432 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36627: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
W20260927 02:53:26.542593 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:26.538315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38673 (local address 127.0.98.62:36627)
0927 02:53:26.538382 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:26.538384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:26.538412 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:53:26.538481 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:26.538482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:26.538513 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:26.538566 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:26.538880 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.539378 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.540059 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.540257 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.540311 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.542526 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38673: BlockingRecv error: recv got EOF from 127.0.98.2:38673 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:53:26.587953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 9076
I20260927 02:53:26.591830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 9228
I20260927 02:53:26.595453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 9360
I20260927 02:53:26.598896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 9006
2026-09-27T02:53:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17291 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T02:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:26Z Disabled control of system clock
I20260927 02:53:26.623454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38185
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:53:26.699806 9556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:26.699995 9556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:26.700012 9556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:26.701138 9556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:26.701169 9556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:26.701179 9556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:26.701190 9556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:26.701200 9556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:26.702450 9556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:26.702646 9556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:26.702910 9556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:26.707631 9564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:26.707803 9561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:26.707700 9556 server_base.cc:1034] running on GCE node
W20260927 02:53:26.707829 9562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:26.708087 9556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:26.708328 9556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:26.709452 9556 hybrid_clock.cc:648] HybridClock initialized: now 1790477606709441 us; error 27 us; skew 500 ppm
I20260927 02:53:26.710620 9556 webserver.cc:460] Webserver started at http://127.0.98.62:38683/ using document root <none> and password file <none>
I20260927 02:53:26.710889 9556 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:26.710937 9556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:26.711040 9556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:26.712047 9556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "149f9349addc4fa7bbb38120c9e1bb29"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "21b8c2ffb2c63147698a8673c98ebf24"
server_key_iv: "ffcf4a029f180b5a7d75db1de71b721a"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.712352 9556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "149f9349addc4fa7bbb38120c9e1bb29"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "21b8c2ffb2c63147698a8673c98ebf24"
server_key_iv: "ffcf4a029f180b5a7d75db1de71b721a"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.713477 9556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:53:26.714105 9570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:26.714243 9556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:26.714320 9556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "149f9349addc4fa7bbb38120c9e1bb29"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "21b8c2ffb2c63147698a8673c98ebf24"
server_key_iv: "ffcf4a029f180b5a7d75db1de71b721a"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.714376 9556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:26.721935 9556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:26.722175 9556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:26.722293 9556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:26.726221 9556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38185
I20260927 02:53:26.726253 9622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38185 every 8 connection(s)
I20260927 02:53:26.726677 9556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:53:26.727751 9623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:26.729830 9623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29: Bootstrap starting.
I20260927 02:53:26.730365 9623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:26.730623 9623 log.cc:824] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:26.731233 9623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29: No bootstrap required, opened a new log
I20260927 02:53:26.732705 9623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149f9349addc4fa7bbb38120c9e1bb29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260927 02:53:26.732834 9623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:26.732856 9623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149f9349addc4fa7bbb38120c9e1bb29, State: Initialized, Role: FOLLOWER
I20260927 02:53:26.732965 9623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149f9349addc4fa7bbb38120c9e1bb29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260927 02:53:26.733034 9623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:53:26.733072 9623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:53:26.733117 9623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:26.733589 9623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149f9349addc4fa7bbb38120c9e1bb29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260927 02:53:26.733671 9623 leader_election.cc:302] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 149f9349addc4fa7bbb38120c9e1bb29; no voters:
I20260927 02:53:26.733807 9623 leader_election.cc:288] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:53:26.733847 9626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:26.733973 9626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [term 1 LEADER]: Becoming Leader. State: Replica: 149f9349addc4fa7bbb38120c9e1bb29, State: Running, Role: LEADER
I20260927 02:53:26.734032 9623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:26.734056 9626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149f9349addc4fa7bbb38120c9e1bb29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260927 02:53:26.734385 9627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "149f9349addc4fa7bbb38120c9e1bb29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149f9349addc4fa7bbb38120c9e1bb29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } } }
I20260927 02:53:26.734392 9628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 149f9349addc4fa7bbb38120c9e1bb29. Latest consensus state: current_term: 1 leader_uuid: "149f9349addc4fa7bbb38120c9e1bb29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149f9349addc4fa7bbb38120c9e1bb29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } } }
I20260927 02:53:26.734498 9627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:26.734560 9628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:26.734961 9635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:26.735414 9635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:53:26.736780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 9556
I20260927 02:53:26.736855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:53:26.736861 9635 catalog_manager.cc:1347] Generated new cluster ID: 88b91109f3e5429f94d041260a40d5fd
I20260927 02:53:26.736901 9635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:53:26.737010 392 external_mini_cluster.cc:1428] Setting key 0b92e8d598ec1b6d43a0ac59e3a4950e
I20260927 02:53:26.742928 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:26.737409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.0.98.62:38185)
0927 02:53:26.737577 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:26.737584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:26.739078 (+ 1494us) server_negotiation.cc:407] Connection header received
0927 02:53:26.739219 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:26.739225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:26.739298 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:26.739421 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:26.739616 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.740509 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.741244 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.741630 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.741740 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.742337 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:26.742341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:26.742341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:53:26.742494 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:26.742526 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:53:26.742538 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:26.742592 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:26.742660 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:53:26.742734 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260927 02:53:26.752377 9635 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:53:26.753005 9635 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:26.763844 9635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 149f9349addc4fa7bbb38120c9e1bb29: Generated new TSK 0
I20260927 02:53:26.763998 9635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:53:26.765918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 02:53:26.847527 9647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:26.847731 9647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:26.847751 9647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:26.848922 9647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:26.848973 9647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:26.848995 9647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:26.850288 9647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:26.850493 9647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:26.850795 9647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:26.855355 9653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:26.855376 9655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:26.855374 9652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:26.855818 9647 server_base.cc:1034] running on GCE node
I20260927 02:53:26.856010 9647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:26.856221 9647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:26.857369 9647 hybrid_clock.cc:648] HybridClock initialized: now 1790477606857359 us; error 25 us; skew 500 ppm
I20260927 02:53:26.858544 9647 webserver.cc:460] Webserver started at http://127.0.98.1:43839/ using document root <none> and password file <none>
I20260927 02:53:26.858783 9647 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:26.858842 9647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:26.859014 9647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:26.860031 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "7d6fd1e7e8584987956eb76cd76382e2"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "ab82e3c45354c8facb523aa6d6c257e3"
server_key_iv: "c6922c8ee3d90a4e66f428e0732741f7"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.860349 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7d6fd1e7e8584987956eb76cd76382e2"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "ab82e3c45354c8facb523aa6d6c257e3"
server_key_iv: "c6922c8ee3d90a4e66f428e0732741f7"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.861574 9647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:26.862246 9661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:26.862396 9647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:26.862471 9647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "7d6fd1e7e8584987956eb76cd76382e2"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "ab82e3c45354c8facb523aa6d6c257e3"
server_key_iv: "c6922c8ee3d90a4e66f428e0732741f7"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.862536 9647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:26.873785 9647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:26.874037 9647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:26.874153 9647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:26.874332 9647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:26.874605 9647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:26.874637 9647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:26.874673 9647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:26.874697 9647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:26.880141 9647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35653
I20260927 02:53:26.880192 9774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35653 every 8 connection(s)
I20260927 02:53:26.880473 9647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:53:26.883502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 9647
I20260927 02:53:26.883585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:53:26.883731 392 external_mini_cluster.cc:1428] Setting key 81a8c9ee797ee2d0e178108cfce87dc9
I20260927 02:53:26.884554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:53:26.885475 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:26.881104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58145 (local address 127.0.98.62:38185)
0927 02:53:26.881161 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:26.881162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:26.881373 (+ 211us) server_negotiation.cc:407] Connection header received
0927 02:53:26.881854 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:26.881855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:26.881866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:26.881914 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:26.882526 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.883053 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.884117 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.884307 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.884360 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.885231 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:26.885232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:26.885233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:53:26.885240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:26.885253 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:53:26.885256 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:26.885300 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:26.885368 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:53:26.885400 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:53:26.885710 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:26.881101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38185 (local address 127.0.98.1:58145)
0927 02:53:26.881272 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:26.881304 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:26.881775 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:26.881980 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:26.881983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:26.881991 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:26.882463 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:26.882467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.883123 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.883124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:26.884047 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:26.884050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.884371 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.884372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:26.884511 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.885102 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:53:26.885103 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:26.885130 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:53:26.885153 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:26.885304 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:26.885308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:26.885311 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:26.885373 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:53:26.885495 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260927 02:53:26.886422 9775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38185
I20260927 02:53:26.886524 9775 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:26.886703 9775 heartbeater.cc:511] Master 127.0.98.62:38185 requested a full tablet report, sending...
I20260927 02:53:26.887240 9587 ts_manager.cc:194] Registered new tserver with Master: 7d6fd1e7e8584987956eb76cd76382e2 (127.0.98.1:35653)
I20260927 02:53:26.887912 9587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58145
W20260927 02:53:26.962234 9778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:26.962420 9778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:26.962440 9778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:26.963603 9778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:26.963641 9778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:26.963665 9778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:26.964993 9778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:26.965185 9778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:26.965428 9778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:26.970082 9783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:26.970093 9786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:26.970093 9784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:26.970306 9778 server_base.cc:1034] running on GCE node
I20260927 02:53:26.970479 9778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:26.970672 9778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:26.971800 9778 hybrid_clock.cc:648] HybridClock initialized: now 1790477606971784 us; error 31 us; skew 500 ppm
I20260927 02:53:26.973024 9778 webserver.cc:460] Webserver started at http://127.0.98.2:41713/ using document root <none> and password file <none>
I20260927 02:53:26.973217 9778 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:26.973263 9778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:26.973380 9778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:26.974375 9778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb009ef4c7c5449b99a969650024b73b"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "9fdbe57bbcb31d39bc7c84bfc122cdc8"
server_key_iv: "f3cb8a4ca57ea5e43ce884ee928fb101"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.974694 9778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb009ef4c7c5449b99a969650024b73b"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "9fdbe57bbcb31d39bc7c84bfc122cdc8"
server_key_iv: "f3cb8a4ca57ea5e43ce884ee928fb101"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.975883 9778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:53:26.976495 9792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:26.976636 9778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:53:26.976711 9778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "eb009ef4c7c5449b99a969650024b73b"
format_stamp: "Formatted at 2026-09-27 02:53:26 on dist-test-slave-mfx9"
server_key: "9fdbe57bbcb31d39bc7c84bfc122cdc8"
server_key_iv: "f3cb8a4ca57ea5e43ce884ee928fb101"
server_key_version: "encryptionkey@0"
I20260927 02:53:26.976768 9778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:26.988440 9778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:26.988689 9778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:26.988806 9778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:26.989006 9778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:26.989285 9778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:26.989317 9778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:26.989346 9778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:26.989367 9778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:26.994575 9778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43715
I20260927 02:53:26.994618 9905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43715 every 8 connection(s)
I20260927 02:53:26.994964 9778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:53:26.999040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 9778
I20260927 02:53:26.999114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:53:26.999264 392 external_mini_cluster.cc:1428] Setting key b5f1cf51969937139656ae95eb08e7e2
I20260927 02:53:26.999770 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:26.995625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56901 (local address 127.0.98.62:38185)
0927 02:53:26.995666 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:26.995668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:26.995868 (+ 200us) server_negotiation.cc:407] Connection header received
0927 02:53:26.996378 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:26.996379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:26.996391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:26.996443 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:26.997081 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.997573 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.998505 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.998692 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.998745 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.999534 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:26.999535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:26.999536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:53:26.999540 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:26.999557 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:53:26.999568 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:26.999616 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:26.999692 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 02:53:26.999714 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:53:26.999998 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:26.995595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38185 (local address 127.0.98.2:56901)
0927 02:53:26.995757 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:26.995788 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:26.996277 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:26.996518 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:26.996522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:26.996532 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:26.996993 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:26.996997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.997650 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.997651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:26.998422 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:26.998424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:26.998771 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:26.998772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:26.998896 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:26.999409 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:53:26.999409 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:26.999443 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:53:26.999464 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:26.999623 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:26.999627 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:26.999630 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:26.999691 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:53:26.999790 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260927 02:53:27.000238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--builtin_ntp_servers=127.0.98.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:53:27.000522 9906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38185
I20260927 02:53:27.000602 9906 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:27.000778 9906 heartbeater.cc:511] Master 127.0.98.62:38185 requested a full tablet report, sending...
I20260927 02:53:27.001101 9587 ts_manager.cc:194] Registered new tserver with Master: eb009ef4c7c5449b99a969650024b73b (127.0.98.2:43715)
I20260927 02:53:27.001443 9587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56901
W20260927 02:53:27.077757 9909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:27.077947 9909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:27.077966 9909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:27.079156 9909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:27.079191 9909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:27.079216 9909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:27.080531 9909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:27.080761 9909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:27.081027 9909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:27.085681 9915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:27.085716 9917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:27.085688 9914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:27.085814 9909 server_base.cc:1034] running on GCE node
I20260927 02:53:27.086117 9909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:27.086333 9909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:27.087464 9909 hybrid_clock.cc:648] HybridClock initialized: now 1790477607087452 us; error 29 us; skew 500 ppm
I20260927 02:53:27.088543 9909 webserver.cc:460] Webserver started at http://127.0.98.3:45761/ using document root <none> and password file <none>
I20260927 02:53:27.088762 9909 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:27.088804 9909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:27.088927 9909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:27.089951 9909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7cb86149a8742d8bbb2ecf363542064"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "50bb52bbaef790e672920ca7c6958e46"
server_key_iv: "5c2d0852f2d85ddddca435b9cca55a05"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.090276 9909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7cb86149a8742d8bbb2ecf363542064"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "50bb52bbaef790e672920ca7c6958e46"
server_key_iv: "5c2d0852f2d85ddddca435b9cca55a05"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.091516 9909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:53:27.092118 9923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.092270 9909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:53:27.092346 9909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "d7cb86149a8742d8bbb2ecf363542064"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "50bb52bbaef790e672920ca7c6958e46"
server_key_iv: "5c2d0852f2d85ddddca435b9cca55a05"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.092406 9909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:27.106693 9909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:27.107064 9909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:27.107191 9909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:27.107386 9909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:27.107681 9909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:27.107714 9909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.107741 9909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:27.107758 9909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.112847 9909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34067
I20260927 02:53:27.112915 10036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34067 every 8 connection(s)
I20260927 02:53:27.113190 9909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:53:27.114634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 9909
I20260927 02:53:27.114725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:53:27.114925 392 external_mini_cluster.cc:1428] Setting key 7a91789184ddbacc58b8268decbfa46c
I20260927 02:53:27.118314 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.113839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46467 (local address 127.0.98.62:38185)
0927 02:53:27.113881 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:27.113882 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:27.114099 (+ 217us) server_negotiation.cc:407] Connection header received
0927 02:53:27.114847 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:27.114848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:27.114860 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:27.114913 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:27.115634 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:27.116182 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:27.117173 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:27.117348 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:27.117396 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:27.118133 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:27.118134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:27.118134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:53:27.118137 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:27.118147 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:53:27.118151 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:27.118187 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:27.118238 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:53:27.118259 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:53:27.118494 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.113825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38185 (local address 127.0.98.3:46467)
0927 02:53:27.113983 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:27.114015 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:27.114675 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:27.114988 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:27.114992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:27.115002 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:27.115567 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:27.115572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:27.116250 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:27.116251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:27.117118 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:27.117120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:27.117405 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:27.117406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:27.117507 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:27.118027 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:53:27.118027 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:27.118054 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:53:27.118078 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:27.118189 (+ 111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:27.118191 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:27.118194 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:27.118237 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:53:27.118311 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260927 02:53:27.118889 10037 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38185
I20260927 02:53:27.118963 10037 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:27.119122 10037 heartbeater.cc:511] Master 127.0.98.62:38185 requested a full tablet report, sending...
I20260927 02:53:27.119422 9587 ts_manager.cc:194] Registered new tserver with Master: d7cb86149a8742d8bbb2ecf363542064 (127.0.98.3:34067)
I20260927 02:53:27.119757 9587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46467
I20260927 02:53:27.126447 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:53:27.128103 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:27.127644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.0.98.62:38185)
0927 02:53:27.127702 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:27.127704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:27.127792 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:53:27.127847 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:27.127848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:27.127859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:27.127900 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:27.128043 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48128: BlockingRecv error: recv got EOF from 127.0.0.1:48128 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:53:27.128140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 9647
I20260927 02:53:27.131824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 9778
I20260927 02:53:27.135286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 9909
I20260927 02:53:27.139379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 9556
2026-09-27T02:53:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (539 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T02:53:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:27Z Disabled control of system clock
I20260927 02:53:27.165596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42061
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:53:27.243110 10048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:27.243301 10048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:27.243320 10048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:27.244412 10048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:27.244452 10048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:27.244463 10048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:27.244474 10048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:27.244484 10048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:27.245770 10048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:27.245983 10048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:27.246235 10048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:27.250612 10056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:27.250640 10054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:27.250654 10053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:27.250721 10048 server_base.cc:1034] running on GCE node
I20260927 02:53:27.251035 10048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:27.251302 10048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:27.252444 10048 hybrid_clock.cc:648] HybridClock initialized: now 1790477607252429 us; error 26 us; skew 500 ppm
I20260927 02:53:27.253562 10048 webserver.cc:460] Webserver started at http://127.0.98.62:41117/ using document root <none> and password file <none>
I20260927 02:53:27.253762 10048 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:27.253808 10048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:27.253935 10048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:27.254979 10048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "60b0647eca8546adb51c54a86ace4d39"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "b3da6f5e5f203bf33bc74c3d2a3ca4a2"
server_key_iv: "6ccd439e55213a6e7127a5e907587504"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.255286 10048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "60b0647eca8546adb51c54a86ace4d39"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "b3da6f5e5f203bf33bc74c3d2a3ca4a2"
server_key_iv: "6ccd439e55213a6e7127a5e907587504"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.256460 10048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:27.257056 10062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.257189 10048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:27.257266 10048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "60b0647eca8546adb51c54a86ace4d39"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "b3da6f5e5f203bf33bc74c3d2a3ca4a2"
server_key_iv: "6ccd439e55213a6e7127a5e907587504"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.257324 10048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:27.260478 10048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:27.260720 10048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:27.260833 10048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:27.264210 10048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42061
I20260927 02:53:27.264237 10114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42061 every 8 connection(s)
I20260927 02:53:27.264513 10048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:53:27.265089 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:27.267601 10115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39: Bootstrap starting.
I20260927 02:53:27.268079 10115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:27.268296 10115 log.cc:824] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:27.268838 10115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39: No bootstrap required, opened a new log
I20260927 02:53:27.270061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 10048
I20260927 02:53:27.270145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:53:27.270308 392 external_mini_cluster.cc:1428] Setting key 99f04574750a11d911ed661700168e88
I20260927 02:53:27.270486 10115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b0647eca8546adb51c54a86ace4d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } }
I20260927 02:53:27.270625 10115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:27.270658 10115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b0647eca8546adb51c54a86ace4d39, State: Initialized, Role: FOLLOWER
I20260927 02:53:27.270814 10115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b0647eca8546adb51c54a86ace4d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } }
I20260927 02:53:27.270902 10115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:53:27.270943 10115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:53:27.270982 10115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:27.271528 10115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b0647eca8546adb51c54a86ace4d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } }
I20260927 02:53:27.271627 10115 leader_election.cc:302] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60b0647eca8546adb51c54a86ace4d39; no voters:
I20260927 02:53:27.271767 10115 leader_election.cc:288] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:53:27.271798 10120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:27.271927 10120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [term 1 LEADER]: Becoming Leader. State: Replica: 60b0647eca8546adb51c54a86ace4d39, State: Running, Role: LEADER
I20260927 02:53:27.271988 10115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:27.272037 10120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b0647eca8546adb51c54a86ace4d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } }
I20260927 02:53:27.272306 10120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60b0647eca8546adb51c54a86ace4d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b0647eca8546adb51c54a86ace4d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } } }
I20260927 02:53:27.272397 10120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:27.272650 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.270785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.0.98.62:42061)
0927 02:53:27.271011 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:27.271021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:27.271054 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:53:27.271192 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:27.271197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:27.271231 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:27.271336 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:27.272080 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:27.272084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:27.272084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:53:27.272089 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:27.272134 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:53:27.272150 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:27.272197 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:27.272333 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:53:27.272425 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260927 02:53:27.272512 10120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60b0647eca8546adb51c54a86ace4d39. Latest consensus state: current_term: 1 leader_uuid: "60b0647eca8546adb51c54a86ace4d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b0647eca8546adb51c54a86ace4d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } } }
I20260927 02:53:27.272966 10120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:27.273273 10126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:27.273712 10126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:53:27.275090 10126 catalog_manager.cc:1347] Generated new cluster ID: aeaba1358d67476a9ed4c6c302e18a7e
I20260927 02:53:27.275144 10126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:53:27.289708 10126 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:53:27.289885 10126 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:27.297719 10126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60b0647eca8546adb51c54a86ace4d39: Generated new TSK 0
I20260927 02:53:27.297911 10126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:53:27.305485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42061
--builtin_ntp_servers=127.0.98.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:53:27.383414 10139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:27.383612 10139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:27.383632 10139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:27.384768 10139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:27.384805 10139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:27.384827 10139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:27.386132 10139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:27.386343 10139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:27.386616 10139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:27.391494 10147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:27.391513 10144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:27.391870 10145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:27.391983 10139 server_base.cc:1034] running on GCE node
I20260927 02:53:27.392162 10139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:27.392385 10139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:27.393522 10139 hybrid_clock.cc:648] HybridClock initialized: now 1790477607393511 us; error 26 us; skew 500 ppm
I20260927 02:53:27.394811 10139 webserver.cc:460] Webserver started at http://127.0.98.1:35969/ using document root <none> and password file <none>
I20260927 02:53:27.395071 10139 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:27.395136 10139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:27.395253 10139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:27.396581 10139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "2dc970b676ab453ca773823182535c79"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "00b74ab7832e110dd26064f61c3559a0"
server_key_iv: "f845df6624b1ce14a43ea1e203db1936"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.397002 10139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2dc970b676ab453ca773823182535c79"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "00b74ab7832e110dd26064f61c3559a0"
server_key_iv: "f845df6624b1ce14a43ea1e203db1936"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.398437 10139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:53:27.399215 10153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.399358 10139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.399442 10139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "2dc970b676ab453ca773823182535c79"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "00b74ab7832e110dd26064f61c3559a0"
server_key_iv: "f845df6624b1ce14a43ea1e203db1936"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.399507 10139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:27.402974 10139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:27.403182 10139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:27.403270 10139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:27.403409 10139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:27.403635 10139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:27.403663 10139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.403681 10139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:27.403698 10139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.408789 10139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42857
I20260927 02:53:27.408857 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42857 every 8 connection(s)
I20260927 02:53:27.409107 10139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:53:27.409931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 10139
I20260927 02:53:27.410017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:53:27.410192 392 external_mini_cluster.cc:1428] Setting key 2a9d609da9043b27f84a4edc361f738a
I20260927 02:53:27.411067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42061
--builtin_ntp_servers=127.0.98.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 02:53:27.411993 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.409738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48075 (local address 127.0.98.62:42061)
0927 02:53:27.409796 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:27.409798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:27.410004 (+ 206us) server_negotiation.cc:407] Connection header received
0927 02:53:27.410569 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:27.410570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:27.410575 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:27.410625 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:27.411771 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:27.411772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:27.411772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:53:27.411777 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:27.411795 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:53:27.411800 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:27.411863 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:27.411911 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:53:27.411926 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:53:27.412225 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.409733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42061 (local address 127.0.98.1:48075)
0927 02:53:27.409896 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:27.409926 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:27.410474 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:27.410690 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:27.410694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:27.410704 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:27.411631 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:53:27.411632 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:27.411659 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:53:27.411686 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:27.411862 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:27.411864 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:27.411866 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:53:27.411913 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:53:27.411957 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260927 02:53:27.412683 10267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42061
I20260927 02:53:27.412772 10267 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:27.412911 10267 heartbeater.cc:511] Master 127.0.98.62:42061 requested a full tablet report, sending...
I20260927 02:53:27.413385 10079 ts_manager.cc:194] Registered new tserver with Master: 2dc970b676ab453ca773823182535c79 (127.0.98.1:42857)
W20260927 02:53:27.488520 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:27.488709 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:27.488729 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:27.490090 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:27.490139 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:27.490175 10270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:27.491654 10270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:27.491833 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:27.492069 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:27.496721 10275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:27.496737 10270 server_base.cc:1034] running on GCE node
W20260927 02:53:27.496722 10278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:27.496800 10276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:27.497193 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:27.497377 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:27.498507 10270 hybrid_clock.cc:648] HybridClock initialized: now 1790477607498492 us; error 30 us; skew 500 ppm
I20260927 02:53:27.504376 10270 webserver.cc:460] Webserver started at http://127.0.98.2:38995/ using document root <none> and password file <none>
I20260927 02:53:27.504577 10270 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:27.504622 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:27.504740 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:27.505764 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b314e81c72940138115ed6bf8db6145"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "dc35e38085c5bdaee5d212457bd934b0"
server_key_iv: "03bc17e3942272075ead4094f2d921f7"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.506073 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b314e81c72940138115ed6bf8db6145"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "dc35e38085c5bdaee5d212457bd934b0"
server_key_iv: "03bc17e3942272075ead4094f2d921f7"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.507268 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 02:53:27.507880 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.508033 10270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.508109 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "5b314e81c72940138115ed6bf8db6145"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "dc35e38085c5bdaee5d212457bd934b0"
server_key_iv: "03bc17e3942272075ead4094f2d921f7"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.508172 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:27.511288 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:27.511507 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:27.511613 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:27.511782 10270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:27.512056 10270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:27.512087 10270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.512115 10270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:27.512136 10270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.517266 10270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34597
I20260927 02:53:27.517316 10397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34597 every 8 connection(s)
I20260927 02:53:27.517590 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:53:27.520220 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.518208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40687 (local address 127.0.98.62:42061)
0927 02:53:27.518266 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:27.518268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:27.518465 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:53:27.518943 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:27.518944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:27.518948 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:27.519008 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:27.520048 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:27.520049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:27.520049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:53:27.520051 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:27.520070 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:53:27.520073 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:27.520112 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:27.520159 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:53:27.520169 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:53:27.520382 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.518188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42061 (local address 127.0.98.2:40687)
0927 02:53:27.518354 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:27.518393 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:27.518862 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:27.519061 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:27.519064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:27.519072 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:27.519903 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:53:27.519904 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:27.519931 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:53:27.519958 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:27.520111 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:27.520113 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:27.520115 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:53:27.520161 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:53:27.520207 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260927 02:53:27.520782 10398 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42061
I20260927 02:53:27.520860 10398 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:27.520995 10398 heartbeater.cc:511] Master 127.0.98.62:42061 requested a full tablet report, sending...
I20260927 02:53:27.521385 10079 ts_manager.cc:194] Registered new tserver with Master: 5b314e81c72940138115ed6bf8db6145 (127.0.98.2:34597)
I20260927 02:53:27.525431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 10270
I20260927 02:53:27.525518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:53:27.525673 392 external_mini_cluster.cc:1428] Setting key f61fc9aaafef9784cff8386f51f31e9a
I20260927 02:53:27.526458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42061
--builtin_ntp_servers=127.0.98.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:53:27.603696 10401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:27.603890 10401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:27.603911 10401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:27.604993 10401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:27.605027 10401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:27.605050 10401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:27.606294 10401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:27.606467 10401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:27.606693 10401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:27.611246 10406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:27.611485 10401 server_base.cc:1034] running on GCE node
W20260927 02:53:27.611263 10407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:27.611248 10409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:27.611769 10401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:27.611999 10401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:27.613133 10401 hybrid_clock.cc:648] HybridClock initialized: now 1790477607613130 us; error 36 us; skew 500 ppm
I20260927 02:53:27.614281 10401 webserver.cc:460] Webserver started at http://127.0.98.3:34533/ using document root <none> and password file <none>
I20260927 02:53:27.614483 10401 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:27.614534 10401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:27.614652 10401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:27.615707 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a8d9d32680240f1b88c4f182eba9ed6"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "185b1c3f2d0927615dc5acae3c4eeb58"
server_key_iv: "b48c817fb2169aaf6d7329487c306aae"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.616040 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a8d9d32680240f1b88c4f182eba9ed6"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "185b1c3f2d0927615dc5acae3c4eeb58"
server_key_iv: "b48c817fb2169aaf6d7329487c306aae"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.617193 10401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:53:27.617810 10415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.617955 10401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:27.618049 10401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "1a8d9d32680240f1b88c4f182eba9ed6"
format_stamp: "Formatted at 2026-09-27 02:53:27 on dist-test-slave-mfx9"
server_key: "185b1c3f2d0927615dc5acae3c4eeb58"
server_key_iv: "b48c817fb2169aaf6d7329487c306aae"
server_key_version: "encryptionkey@0"
I20260927 02:53:27.618112 10401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:27.621331 10401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:27.621565 10401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:27.621680 10401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:27.621860 10401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:27.622145 10401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:27.622177 10401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.622202 10401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:27.622225 10401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:27.627449 10401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32927
I20260927 02:53:27.627504 10528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32927 every 8 connection(s)
I20260927 02:53:27.627760 10401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:53:27.630484 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.628432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38079 (local address 127.0.98.62:42061)
0927 02:53:27.628486 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:27.628488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:27.628712 (+ 224us) server_negotiation.cc:407] Connection header received
0927 02:53:27.629168 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:27.629170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:27.629176 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:27.629224 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:27.630238 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:27.630239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:27.630240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:53:27.630244 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:27.630259 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:53:27.630263 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:27.630313 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:27.630400 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 02:53:27.630414 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:53:27.630494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 10401
I20260927 02:53:27.630571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:53:27.630666 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:27.628412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42061 (local address 127.0.98.3:38079)
0927 02:53:27.628602 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:27.628635 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:27.629077 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:27.629300 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:27.629304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:27.629313 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:27.630105 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:53:27.630106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:27.630131 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:53:27.630157 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:27.630315 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:27.630319 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:27.630322 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:27.630404 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:53:27.630462 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260927 02:53:27.630730 392 external_mini_cluster.cc:1428] Setting key 3271361507230d4b77ef86841664c172
I20260927 02:53:27.631107 10529 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42061
I20260927 02:53:27.631187 10529 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:27.631321 10529 heartbeater.cc:511] Master 127.0.98.62:42061 requested a full tablet report, sending...
I20260927 02:53:27.631634 10079 ts_manager.cc:194] Registered new tserver with Master: 1a8d9d32680240f1b88c4f182eba9ed6 (127.0.98.3:32927)
I20260927 02:53:27.642120 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:53:27.643756 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:53:27.643370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.0.98.62:42061)
0927 02:53:27.643424 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:27.643426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:27.643528 (+ 102us) server_negotiation.cc:407] Connection header received
0927 02:53:27.643571 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:27.643571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:27.643575 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:27.643612 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:27.643711 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55068: BlockingRecv error: recv got EOF from 127.0.0.1:55068 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:53:27.643838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 10139
I20260927 02:53:27.647619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 10270
I20260927 02:53:27.651118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 10401
I20260927 02:53:27.654337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 10048
2026-09-27T02:53:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (514 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:53:27 dist-test-slave-mfx9 krb5kdc[10539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:53:27 dist-test-slave-mfx9 krb5kdc[10539](info): set up 2 sockets
Sep 27 02:53:27 dist-test-slave-mfx9 krb5kdc[10539](info): commencing operation
krb5kdc: starting...
W20260927 02:53:28.599216 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:53:28 dist-test-slave-mfx9 krb5kdc[10539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:53:28.707270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46193
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:28.784186 10552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:28.784366 10552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:28.784386 10552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:28.785486 10552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:53:28.785522 10552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:28.785534 10552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:28.785557 10552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:53:28.785578 10552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:53:28.787010 10552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:28.787228 10552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:28.787428 10552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:28.791841 10557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:28.791896 10558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:28.791941 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:28.792037 10552 server_base.cc:1034] running on GCE node
I20260927 02:53:28.792259 10552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:28.792513 10552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:28.793648 10552 hybrid_clock.cc:648] HybridClock initialized: now 1790477608793634 us; error 30 us; skew 500 ppm
Sep 27 02:53:28 dist-test-slave-mfx9 krb5kdc[10539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:28.795879 10552 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:53:28.796294 10552 webserver.cc:460] Webserver started at http://127.0.98.62:43773/ using document root <none> and password file <none>
I20260927 02:53:28.796494 10552 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:28.796537 10552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:28.796655 10552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:28.797621 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "a161ac4b59134cd7902fcc9786e99862"
format_stamp: "Formatted at 2026-09-27 02:53:28 on dist-test-slave-mfx9"
server_key: "0cd8ceb55b0269172850f1635ccc0dfb"
server_key_iv: "a37c0a7c3718fd87ce27fe99465d5762"
server_key_version: "encryptionkey@0"
I20260927 02:53:28.797931 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "a161ac4b59134cd7902fcc9786e99862"
format_stamp: "Formatted at 2026-09-27 02:53:28 on dist-test-slave-mfx9"
server_key: "0cd8ceb55b0269172850f1635ccc0dfb"
server_key_iv: "a37c0a7c3718fd87ce27fe99465d5762"
server_key_version: "encryptionkey@0"
I20260927 02:53:28.799122 10552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:28.799734 10567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:28.799872 10552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:28.799950 10552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "a161ac4b59134cd7902fcc9786e99862"
format_stamp: "Formatted at 2026-09-27 02:53:28 on dist-test-slave-mfx9"
server_key: "0cd8ceb55b0269172850f1635ccc0dfb"
server_key_iv: "a37c0a7c3718fd87ce27fe99465d5762"
server_key_version: "encryptionkey@0"
I20260927 02:53:28.800004 10552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:28.808012 10552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:28.809541 10552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:28.809659 10552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:28.813207 10552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46193
I20260927 02:53:28.813235 10619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46193 every 8 connection(s)
I20260927 02:53:28.813532 10552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:53:28.814388 10620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:28.816527 10620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862: Bootstrap starting.
I20260927 02:53:28.817026 10620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:28.817237 10620 log.cc:824] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:28.817790 10620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862: No bootstrap required, opened a new log
I20260927 02:53:28.819298 10620 raft_consensus.cc:348] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161ac4b59134cd7902fcc9786e99862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260927 02:53:28.819427 10620 raft_consensus.cc:374] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:28.819448 10620 raft_consensus.cc:729] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a161ac4b59134cd7902fcc9786e99862, State: Initialized, Role: FOLLOWER
I20260927 02:53:28.819566 10620 consensus_queue.cc:260] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161ac4b59134cd7902fcc9786e99862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260927 02:53:28.819615 10620 raft_consensus.cc:388] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:53:28.819633 10620 raft_consensus.cc:482] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:53:28.819661 10620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:28.820322 10620 raft_consensus.cc:504] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161ac4b59134cd7902fcc9786e99862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260927 02:53:28.820400 10620 leader_election.cc:302] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a161ac4b59134cd7902fcc9786e99862; no voters:
I20260927 02:53:28.820513 10620 leader_election.cc:288] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:53:28.820544 10623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:28.820712 10623 raft_consensus.cc:686] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [term 1 LEADER]: Becoming Leader. State: Replica: a161ac4b59134cd7902fcc9786e99862, State: Running, Role: LEADER
I20260927 02:53:28.820732 10620 sys_catalog.cc:564] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:53:28.820797 10623 consensus_queue.cc:237] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161ac4b59134cd7902fcc9786e99862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260927 02:53:28.821234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 10552
I20260927 02:53:28.821321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:53:28.821488 392 external_mini_cluster.cc:1428] Setting key 26f2e49f7128433d027adb4976e627d1
I20260927 02:53:28.821867 10624 sys_catalog.cc:455] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a161ac4b59134cd7902fcc9786e99862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161ac4b59134cd7902fcc9786e99862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } } }
I20260927 02:53:28.821978 10624 sys_catalog.cc:458] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:28.822130 10625 sys_catalog.cc:455] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a161ac4b59134cd7902fcc9786e99862. Latest consensus state: current_term: 1 leader_uuid: "a161ac4b59134cd7902fcc9786e99862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161ac4b59134cd7902fcc9786e99862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } } }
I20260927 02:53:28.822347 10641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:53:28.822355 10625 sys_catalog.cc:458] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862 [sys.catalog]: This master's current role is: LEADER
I20260927 02:53:28.822867 10641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:53:28.824254 10641 catalog_manager.cc:1347] Generated new cluster ID: bff0f75a517b46e5bcfc795cc2feca66
I20260927 02:53:28.824301 10641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:53:28 dist-test-slave-mfx9 krb5kdc[10539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:28.832347 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:28.821890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.0.98.62:46193)
0927 02:53:28.822065 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:28.822074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:28.822266 (+ 192us) server_negotiation.cc:407] Connection header received
0927 02:53:28.822412 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:28.822419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:28.822485 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:28.822643 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:28.823357 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:28.824513 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:28.825532 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:28.826295 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:28.826384 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:28.829056 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:28.829060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:28.829060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:28.829072 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:28.830875 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:28.831242 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:28.831245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:28.831245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:28.831349 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:28.831526 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:28.831527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:28.831527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:28.831789 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:28.831922 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:28.832031 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 02:53:28.832151 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260927 02:53:28.839953 10641 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:53:28.840428 10641 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:53:28.845881 10641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862: Generated new TSK 0
I20260927 02:53:28.846037 10641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:53:28.873615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:28.951175 10646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:28.951363 10646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:28.951386 10646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:28.952453 10646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:28.952488 10646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:28.952512 10646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:53:28.953930 10646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:28.954094 10646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:28.954262 10646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:28.958698 10654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:28.958838 10646 server_base.cc:1034] running on GCE node
W20260927 02:53:28.958706 10652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:28.958734 10651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:28.959136 10646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:28.959357 10646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:28.960490 10646 hybrid_clock.cc:648] HybridClock initialized: now 1790477608960476 us; error 30 us; skew 500 ppm
Sep 27 02:53:28 dist-test-slave-mfx9 krb5kdc[10539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:28.962778 10646 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:53:28.963183 10646 webserver.cc:460] Webserver started at http://127.0.98.1:41305/ using document root <none> and password file <none>
I20260927 02:53:28.963366 10646 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:28.963410 10646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:28.963536 10646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:28.964483 10646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a"
format_stamp: "Formatted at 2026-09-27 02:53:28 on dist-test-slave-mfx9"
server_key: "c62127116bd1ac0894fc774bce89648b"
server_key_iv: "d022d21c13a244aeb2993b82e9b8da3a"
server_key_version: "encryptionkey@0"
I20260927 02:53:28.964799 10646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a"
format_stamp: "Formatted at 2026-09-27 02:53:28 on dist-test-slave-mfx9"
server_key: "c62127116bd1ac0894fc774bce89648b"
server_key_iv: "d022d21c13a244aeb2993b82e9b8da3a"
server_key_version: "encryptionkey@0"
I20260927 02:53:28.965940 10646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:28.966584 10661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:28.966740 10646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:28.966840 10646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a"
format_stamp: "Formatted at 2026-09-27 02:53:28 on dist-test-slave-mfx9"
server_key: "c62127116bd1ac0894fc774bce89648b"
server_key_iv: "d022d21c13a244aeb2993b82e9b8da3a"
server_key_version: "encryptionkey@0"
I20260927 02:53:28.966902 10646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:28.976101 10646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:28.977699 10646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:28.977818 10646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:28.978024 10646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:28.978284 10646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:28.978314 10646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:28.978340 10646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:28.978364 10646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:28.983582 10646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34153
I20260927 02:53:28.983631 10774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34153 every 8 connection(s)
I20260927 02:53:28.983896 10646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:53:28.987689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 10646
I20260927 02:53:28.987774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:53:28.987920 392 external_mini_cluster.cc:1428] Setting key ec0b0d3b41fb8622bed65d61e4a34ea1
Sep 27 02:53:28 dist-test-slave-mfx9 krb5kdc[10539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:28.992641 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:28.984517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44371 (local address 127.0.98.62:46193)
0927 02:53:28.984569 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:28.984571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:28.984744 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:53:28.985222 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:28.985223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:28.985236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:28.985299 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:28.986212 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:28.986707 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:28.987726 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:28.987989 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:28.988051 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:28.990019 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:28.990021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:28.990021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:28.990029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:28.991357 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:28.991795 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:28.991796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:28.991796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:28.991841 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:28.992078 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:28.992079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:28.992079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:28.992167 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:28.992209 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:28.992522 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 02:53:28.992581 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:53:28.992880 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:28.984483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46193 (local address 127.0.98.1:44371)
0927 02:53:28.984634 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:28.984667 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:28.985127 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:28.985369 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:28.985373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:28.985930 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:28.986114 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:28.986119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:28.986842 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:28.986843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:28.987646 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:28.987649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:28.988057 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:28.988058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:28.988172 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:28.988671 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:28.988679 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:28.989936 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:28.991438 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:28.991445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:28.991448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:28.991722 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:28.991891 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:28.991892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:28.991892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:28.992018 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:28.992220 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:28.992225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:28.992395 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 02:53:28.992506 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 02:53:28.992662 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260927 02:53:28.993420 10775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46193
I20260927 02:53:28.993515 10775 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:28.993692 10775 heartbeater.cc:511] Master 127.0.98.62:46193 requested a full tablet report, sending...
I20260927 02:53:28.994148 10584 ts_manager.cc:194] Registered new tserver with Master: 7cdae604c02f4b0e83bce5b76c4c9e6a (127.0.98.1:34153)
I20260927 02:53:28.994717 10584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44371
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:53:29.011334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:29.088611 10780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:29.088806 10780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:29.088836 10780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:29.089934 10780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:29.089972 10780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:29.090005 10780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:53:29.091403 10780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:29.091636 10780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:29.091832 10780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:29.096467 10785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:29.096474 10786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:29.096480 10788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:29.096946 10780 server_base.cc:1034] running on GCE node
I20260927 02:53:29.097143 10780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:29.097347 10780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:29.098495 10780 hybrid_clock.cc:648] HybridClock initialized: now 1790477609098477 us; error 26 us; skew 500 ppm
Sep 27 02:53:29 dist-test-slave-mfx9 krb5kdc[10539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:29.100867 10780 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:53:29.101310 10780 webserver.cc:460] Webserver started at http://127.0.98.2:43463/ using document root <none> and password file <none>
I20260927 02:53:29.101517 10780 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:29.101560 10780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:29.101678 10780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:29.102689 10780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "2d89980b63ca42b3acffb748b2340a1e"
format_stamp: "Formatted at 2026-09-27 02:53:29 on dist-test-slave-mfx9"
server_key: "8badc4752164536ada699fd9f381895e"
server_key_iv: "255cd67fb061af4723c1d050a51e82f0"
server_key_version: "encryptionkey@0"
I20260927 02:53:29.103091 10780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2d89980b63ca42b3acffb748b2340a1e"
format_stamp: "Formatted at 2026-09-27 02:53:29 on dist-test-slave-mfx9"
server_key: "8badc4752164536ada699fd9f381895e"
server_key_iv: "255cd67fb061af4723c1d050a51e82f0"
server_key_version: "encryptionkey@0"
I20260927 02:53:29.104295 10780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:53:29.104964 10795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:29.105121 10780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:53:29.105195 10780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "2d89980b63ca42b3acffb748b2340a1e"
format_stamp: "Formatted at 2026-09-27 02:53:29 on dist-test-slave-mfx9"
server_key: "8badc4752164536ada699fd9f381895e"
server_key_iv: "255cd67fb061af4723c1d050a51e82f0"
server_key_version: "encryptionkey@0"
I20260927 02:53:29.105244 10780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:29.114019 10780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:29.115679 10780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:29.115780 10780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:29.115943 10780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:29.116170 10780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:29.116197 10780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:29.116216 10780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:29.116232 10780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:29.121527 10780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41831
I20260927 02:53:29.121595 10908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41831 every 8 connection(s)
I20260927 02:53:29.121848 10780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:53:29.125214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 10780
I20260927 02:53:29.125294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:53:29.125434 392 external_mini_cluster.cc:1428] Setting key a187ee5f0b4e7940f043b5f3d9aba374
Sep 27 02:53:29 dist-test-slave-mfx9 krb5kdc[10539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:29.130569 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.122482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46579 (local address 127.0.98.62:46193)
0927 02:53:29.122563 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.122565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.122718 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:53:29.123205 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.123206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.123216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.123260 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:29.124095 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.124577 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.125525 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.125792 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.125853 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.128069 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:29.128070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:29.128071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:29.128077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:29.129322 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:29.129738 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:29.129739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:29.129739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:29.129782 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:29.130026 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:29.130026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:29.130027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:29.130117 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:29.130162 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.130452 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.130509 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:53:29.130844 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.122472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46193 (local address 127.0.98.2:46579)
0927 02:53:29.122609 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:29.122644 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:29.123124 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:29.123320 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:29.123323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:29.123844 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:29.124029 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.124034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.124635 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.124636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.125445 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.125447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.125865 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.125866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.126002 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.126466 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:29.126476 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:29.127987 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:29.129391 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:29.129397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:29.129400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:29.129671 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:29.129832 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:29.129833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:29.129833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:29.129957 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:29.130170 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:29.130175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:29.130331 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 02:53:29.130445 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:53:29.130600 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260927 02:53:29.131335 10909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46193
I20260927 02:53:29.131419 10909 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:29.131587 10909 heartbeater.cc:511] Master 127.0.98.62:46193 requested a full tablet report, sending...
I20260927 02:53:29.131913 10584 ts_manager.cc:194] Registered new tserver with Master: 2d89980b63ca42b3acffb748b2340a1e (127.0.98.2:41831)
I20260927 02:53:29.132256 10584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46579
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:53:29.148723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:53:29.227476 10914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:53:29.227669 10914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:53:29.227690 10914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:53:29.228752 10914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:53:29.228782 10914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:53:29.228806 10914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:53:29.230127 10914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:53:29.230293 10914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:53:29.230455 10914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:53:29.234967 10920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:29.234990 10919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:53:29.234973 10922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:53:29.235131 10914 server_base.cc:1034] running on GCE node
I20260927 02:53:29.235355 10914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:53:29.235563 10914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:53:29.236693 10914 hybrid_clock.cc:648] HybridClock initialized: now 1790477609236683 us; error 27 us; skew 500 ppm
Sep 27 02:53:29 dist-test-slave-mfx9 krb5kdc[10539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:53:29.239015 10914 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:53:29.239408 10914 webserver.cc:460] Webserver started at http://127.0.98.3:33871/ using document root <none> and password file <none>
I20260927 02:53:29.239591 10914 fs_manager.cc:359] Metadata directory not provided
I20260927 02:53:29.239637 10914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:53:29.239756 10914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:53:29.240741 10914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "ba4abe054136406cb5d7fe5c69b7be15"
format_stamp: "Formatted at 2026-09-27 02:53:29 on dist-test-slave-mfx9"
server_key: "dd36ccdb08f06ca6443355ffb0a9c199"
server_key_iv: "3b72131873bc1c7f0b50cf5fc8a70614"
server_key_version: "encryptionkey@0"
I20260927 02:53:29.241063 10914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ba4abe054136406cb5d7fe5c69b7be15"
format_stamp: "Formatted at 2026-09-27 02:53:29 on dist-test-slave-mfx9"
server_key: "dd36ccdb08f06ca6443355ffb0a9c199"
server_key_iv: "3b72131873bc1c7f0b50cf5fc8a70614"
server_key_version: "encryptionkey@0"
I20260927 02:53:29.242204 10914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:53:29.242825 10929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:29.242947 10914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:29.243038 10914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "ba4abe054136406cb5d7fe5c69b7be15"
format_stamp: "Formatted at 2026-09-27 02:53:29 on dist-test-slave-mfx9"
server_key: "dd36ccdb08f06ca6443355ffb0a9c199"
server_key_iv: "3b72131873bc1c7f0b50cf5fc8a70614"
server_key_version: "encryptionkey@0"
I20260927 02:53:29.243099 10914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:53:29.251720 10914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:53:29.253623 10914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:53:29.253742 10914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:53:29.253921 10914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:53:29.254196 10914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:53:29.254228 10914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:29.254256 10914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:53:29.254279 10914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:53:29.259347 10914 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40035
I20260927 02:53:29.259399 11042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40035 every 8 connection(s)
I20260927 02:53:29.259658 10914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:53:29.262528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 10914
I20260927 02:53:29.262632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:53:29.262807 392 external_mini_cluster.cc:1428] Setting key f71ce6f122da468c6e197fd59a83ebb3
Sep 27 02:53:29 dist-test-slave-mfx9 krb5kdc[10539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:53:29.267902 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.260274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37115 (local address 127.0.98.62:46193)
0927 02:53:29.260331 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.260333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.260506 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:53:29.260995 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.260997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.261008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.261058 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:29.261943 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.262475 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.263796 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.263978 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.264030 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.265786 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:29.265787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:29.265787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:29.265794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:29.266966 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:29.267260 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:29.267261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:29.267261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:29.267303 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:29.267473 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:29.267473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:29.267473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:29.267559 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:29.267604 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.267790 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.267841 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:53:29.268066 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.260247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46193 (local address 127.0.98.3:37115)
0927 02:53:29.260399 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:29.260431 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:29.260899 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:29.261131 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:29.261136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:29.261670 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:53:29.261850 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.261854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.262560 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.262561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.263730 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.263733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.264036 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.264037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.264129 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.264497 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:53:29.264517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:53:29.265713 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:53:29.267027 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:29.267031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:29.267032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:29.267212 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:29.267348 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:53:29.267348 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:53:29.267349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:53:29.267431 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:53:29.267601 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:53:29.267603 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:53:29.267702 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 02:53:29.267773 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:53:29.267877 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260927 02:53:29.268448 11043 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46193
I20260927 02:53:29.268525 11043 heartbeater.cc:464] Registering TS with master...
I20260927 02:53:29.268692 11043 heartbeater.cc:511] Master 127.0.98.62:46193 requested a full tablet report, sending...
I20260927 02:53:29.269006 10584 ts_manager.cc:194] Registered new tserver with Master: ba4abe054136406cb5d7fe5c69b7be15 (127.0.98.3:40035)
I20260927 02:53:29.269351 10584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37115
I20260927 02:53:29.274209 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:53:29.280392 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.275348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.0.98.62:46193)
0927 02:53:29.275401 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.275403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.275497 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:53:29.275560 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.275560 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.275569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.275609 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:53:29.276101 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.276547 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.277219 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.277414 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.277465 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.278446 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:53:29.278447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:53:29.278448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:53:29.278454 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:53:29.279439 (+ 985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:29.279718 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:29.279718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:29.279719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:29.279766 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:53:29.279922 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:53:29.279923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:53:29.279923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:53:29.280056 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:53:29.280143 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.280244 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.280310 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:53:29.281556 10584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:53:29.283017 10584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:53:29.288607 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.284984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41831 (local address 127.0.0.1:40698)
0927 02:53:29.285242 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:29.285250 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:29.285288 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:29.285622 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:29.285623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:29.285624 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:29.285667 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.285669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.286853 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.286854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.287638 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.287641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.288289 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.288289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.288362 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.288375 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:53:29.288460 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:53:29.288522 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":68,"threads_started":1}
I20260927 02:53:29.288638 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.284903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.0.98.1:34153)
0927 02:53:29.285141 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.285146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.285157 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:53:29.285196 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.285198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.285213 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.285291 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:29.285486 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.286334 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.287656 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.288342 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.288410 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.288458 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.288521 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.288556 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
I20260927 02:53:29.288911 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.285088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.0.98.2:41831)
0927 02:53:29.285412 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.285415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.285423 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:29.285455 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.285457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.285469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.285547 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:29.285737 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.286781 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.287716 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.288211 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.288272 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.288328 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.288790 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.288832 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":81,"threads_started":1}
I20260927 02:53:29.288998 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.285132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40035 (local address 127.0.0.1:44480)
0927 02:53:29.285256 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:29.285265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:29.285312 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:29.285522 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:29.285522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:29.285523 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:29.285551 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.285551 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.286777 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.286778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.287946 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.287948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.288828 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.288828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.288871 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.288874 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:29.288912 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 02:53:29.288938 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260927 02:53:29.289083 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.285220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.0.98.3:40035)
0927 02:53:29.285390 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.285394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.285403 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:53:29.285438 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.285440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.285453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.285529 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:29.285590 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.286656 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.288026 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.288693 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.288754 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.288802 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.288960 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.289010 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260927 02:53:29.288609 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.284799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34153 (local address 127.0.0.1:55708)
0927 02:53:29.284930 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:29.284970 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:29.285041 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:29.285372 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:29.285374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:29.285376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:29.285421 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.285422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.286418 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.286419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.287566 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.287569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.288409 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.288409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.288461 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.288464 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:53:29.288509 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:53:29.288537 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260927 02:53:29.289651 10977 tablet_service.cc:1467] Processing CreateTablet for tablet 862a1b32997449d8ac7056a8d004f58d (DEFAULT_TABLE table=test-table [id=851bcbd2da2543fc89f84d9c57accfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:29.289650 10843 tablet_service.cc:1467] Processing CreateTablet for tablet 862a1b32997449d8ac7056a8d004f58d (DEFAULT_TABLE table=test-table [id=851bcbd2da2543fc89f84d9c57accfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:29.289963 10977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862a1b32997449d8ac7056a8d004f58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:29.289964 10843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862a1b32997449d8ac7056a8d004f58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:29.293058 11058 tablet_bootstrap.cc:492] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Bootstrap starting.
I20260927 02:53:29.293175 10709 tablet_service.cc:1467] Processing CreateTablet for tablet 862a1b32997449d8ac7056a8d004f58d (DEFAULT_TABLE table=test-table [id=851bcbd2da2543fc89f84d9c57accfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:53:29.293514 10709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862a1b32997449d8ac7056a8d004f58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:53:29.293704 11058 tablet_bootstrap.cc:654] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:29.294245 11058 log.cc:824] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:29.294595 11057 tablet_bootstrap.cc:492] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: Bootstrap starting.
I20260927 02:53:29.294950 11058 tablet_bootstrap.cc:492] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: No bootstrap required, opened a new log
I20260927 02:53:29.295008 11058 ts_tablet_manager.cc:1397] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:53:29.295233 11057 tablet_bootstrap.cc:654] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:29.295747 11057 log.cc:824] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:29.296412 11058 raft_consensus.cc:348] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.296514 11061 tablet_bootstrap.cc:492] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a: Bootstrap starting.
I20260927 02:53:29.296564 11058 raft_consensus.cc:374] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:29.296610 11058 raft_consensus.cc:729] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d89980b63ca42b3acffb748b2340a1e, State: Initialized, Role: FOLLOWER
I20260927 02:53:29.296725 11058 consensus_queue.cc:260] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.296975 11058 ts_tablet_manager.cc:1428] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:53:29.296991 10909 heartbeater.cc:503] Master 127.0.98.62:46193 was elected leader, sending a full tablet report...
I20260927 02:53:29.297057 11061 tablet_bootstrap.cc:654] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a: Neither blocks nor log segments found. Creating new log.
I20260927 02:53:29.297499 11057 tablet_bootstrap.cc:492] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: No bootstrap required, opened a new log
I20260927 02:53:29.297488 11061 log.cc:824] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:53:29.297550 11057 ts_tablet_manager.cc:1397] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:53:29.298090 11061 tablet_bootstrap.cc:492] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a: No bootstrap required, opened a new log
I20260927 02:53:29.298149 11061 ts_tablet_manager.cc:1397] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:53:29.299072 11057 raft_consensus.cc:348] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.299233 11057 raft_consensus.cc:374] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:29.299269 11057 raft_consensus.cc:729] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4abe054136406cb5d7fe5c69b7be15, State: Initialized, Role: FOLLOWER
I20260927 02:53:29.299371 11057 consensus_queue.cc:260] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.299603 11057 ts_tablet_manager.cc:1428] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:53:29.299620 11043 heartbeater.cc:503] Master 127.0.98.62:46193 was elected leader, sending a full tablet report...
I20260927 02:53:29.299757 11061 raft_consensus.cc:348] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.299896 11061 raft_consensus.cc:374] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:53:29.299938 11061 raft_consensus.cc:729] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdae604c02f4b0e83bce5b76c4c9e6a, State: Initialized, Role: FOLLOWER
I20260927 02:53:29.300034 11061 consensus_queue.cc:260] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.300324 10775 heartbeater.cc:503] Master 127.0.98.62:46193 was elected leader, sending a full tablet report...
I20260927 02:53:29.300408 11061 ts_tablet_manager.cc:1428] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260927 02:53:29.372339 10910 tablet.cc:2369] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:53:29.387177 11062 raft_consensus.cc:482] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:53:29.387293 11062 raft_consensus.cc:504] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.387630 11062 leader_election.cc:288] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cdae604c02f4b0e83bce5b76c4c9e6a (127.0.98.1:34153), ba4abe054136406cb5d7fe5c69b7be15 (127.0.98.3:40035)
I20260927 02:53:29.391160 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.388020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59803 (local address 127.0.98.3:40035)
0927 02:53:29.388103 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.388106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.388151 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:53:29.388210 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.388211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.388222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.388272 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:29.388364 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.389019 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.390253 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.390833 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.390900 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.390909 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.391047 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.391090 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 02:53:29.391160 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.387867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54151 (local address 127.0.98.1:34153)
0927 02:53:29.387921 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.387923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.387931 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:29.387956 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.387957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.387967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.388019 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:53:29.388183 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.388927 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.390254 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.390833 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.390900 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.390910 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.391059 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.391094 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:53:29.391170 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.387777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34153 (local address 127.0.98.2:54151)
0927 02:53:29.387828 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:29.387842 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:29.387894 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:29.388047 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:29.388048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:29.388051 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:29.388120 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.388122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.388992 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.388993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.390175 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.390177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.390901 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.390901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.390986 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.390988 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:53:29.391051 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:53:29.391078 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":9}
I20260927 02:53:29.391175 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.387951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40035 (local address 127.0.98.2:59803)
0927 02:53:29.388098 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:29.388107 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:53:29.388161 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:53:29.388272 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:53:29.388272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:53:29.388273 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:53:29.388315 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.388316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.389084 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.389084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.390175 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:53:29.390178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.390901 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.390901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:53:29.390978 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.390982 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:53:29.391038 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:53:29.391077 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":6,"thread_start_us":87,"threads_started":1}
I20260927 02:53:29.391513 10729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862a1b32997449d8ac7056a8d004f58d" candidate_uuid: "2d89980b63ca42b3acffb748b2340a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" is_pre_election: true
I20260927 02:53:29.391628 10729 raft_consensus.cc:2454] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d89980b63ca42b3acffb748b2340a1e in term 0.
I20260927 02:53:29.391795 10797 leader_election.cc:302] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d89980b63ca42b3acffb748b2340a1e, 7cdae604c02f4b0e83bce5b76c4c9e6a; no voters:
I20260927 02:53:29.391908 11062 raft_consensus.cc:2781] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:53:29.391906 10997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862a1b32997449d8ac7056a8d004f58d" candidate_uuid: "2d89980b63ca42b3acffb748b2340a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4abe054136406cb5d7fe5c69b7be15" is_pre_election: true
I20260927 02:53:29.391975 11062 raft_consensus.cc:482] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:53:29.391999 11062 raft_consensus.cc:3037] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:29.392035 10997 raft_consensus.cc:2454] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d89980b63ca42b3acffb748b2340a1e in term 0.
I20260927 02:53:29.392681 11062 raft_consensus.cc:504] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.392776 11062 leader_election.cc:288] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [CANDIDATE]: Term 1 election: Requested vote from peers 7cdae604c02f4b0e83bce5b76c4c9e6a (127.0.98.1:34153), ba4abe054136406cb5d7fe5c69b7be15 (127.0.98.3:40035)
I20260927 02:53:29.392915 10729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862a1b32997449d8ac7056a8d004f58d" candidate_uuid: "2d89980b63ca42b3acffb748b2340a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a"
I20260927 02:53:29.392947 10997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862a1b32997449d8ac7056a8d004f58d" candidate_uuid: "2d89980b63ca42b3acffb748b2340a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4abe054136406cb5d7fe5c69b7be15"
I20260927 02:53:29.392979 10729 raft_consensus.cc:3037] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:29.393005 10997 raft_consensus.cc:3037] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:53:29.393625 10997 raft_consensus.cc:2454] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d89980b63ca42b3acffb748b2340a1e in term 1.
I20260927 02:53:29.393672 10729 raft_consensus.cc:2454] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d89980b63ca42b3acffb748b2340a1e in term 1.
I20260927 02:53:29.393791 10797 leader_election.cc:302] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d89980b63ca42b3acffb748b2340a1e, 7cdae604c02f4b0e83bce5b76c4c9e6a; no voters:
I20260927 02:53:29.393875 11062 raft_consensus.cc:2781] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:53:29.394017 11062 raft_consensus.cc:686] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 1 LEADER]: Becoming Leader. State: Replica: 2d89980b63ca42b3acffb748b2340a1e, State: Running, Role: LEADER
I20260927 02:53:29.394097 11062 consensus_queue.cc:237] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260927 02:53:29.394878 10583 catalog_manager.cc:5495] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d89980b63ca42b3acffb748b2340a1e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2d89980b63ca42b3acffb748b2340a1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d89980b63ca42b3acffb748b2340a1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } health_report { overall_health: UNKNOWN } } }
I20260927 02:53:29.403774 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.401699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.0.98.2:41831)
0927 02:53:29.401750 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:53:29.401751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:53:29.401759 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:53:29.401811 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:53:29.401811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:53:29.401825 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:53:29.401876 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:53:29.401963 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.402443 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.403188 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:53:29.403358 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:53:29.403404 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:53:29.403551 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:53:29.403638 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:53:29.403688 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:53:29.403699 (+ 11us) server_negotiation.cc:299] Negotiation successful
0927 02:53:29.403714 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:53:29.405193 10997 raft_consensus.cc:1261] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 1 FOLLOWER]: Refusing update from remote peer 2d89980b63ca42b3acffb748b2340a1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:29.405206 10729 raft_consensus.cc:1261] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 1 FOLLOWER]: Refusing update from remote peer 2d89980b63ca42b3acffb748b2340a1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:53:29.405481 11062 consensus_queue.cc:1035] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cdae604c02f4b0e83bce5b76c4c9e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:29.405614 11067 consensus_queue.cc:1035] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4abe054136406cb5d7fe5c69b7be15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:53:29.408707 11072 mvcc.cc:205] Tried to move back new op lower bound from 7333796288121651200 to 7333796288078680064. Current Snapshot: MvccSnapshot[applied={T|T < 7333796288121651200 or (T in {7333796288121651200})}]
I20260927 02:53:29.409926 10584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:29.410015 10584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:53:29.410643 10584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a161ac4b59134cd7902fcc9786e99862: Sending DeleteTablet for 3 replicas of tablet 862a1b32997449d8ac7056a8d004f58d
I20260927 02:53:29.410943 10709 tablet_service.cc:1514] Processing DeleteTablet for tablet 862a1b32997449d8ac7056a8d004f58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55708
I20260927 02:53:29.410965 10843 tablet_service.cc:1514] Processing DeleteTablet for tablet 862a1b32997449d8ac7056a8d004f58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40698
I20260927 02:53:29.411049 10977 tablet_service.cc:1514] Processing DeleteTablet for tablet 862a1b32997449d8ac7056a8d004f58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:53:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44480
I20260927 02:53:29.411170 11081 tablet_replica.cc:330] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a: stopping tablet replica
I20260927 02:53:29.411259 11081 raft_consensus.cc:2229] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:29.411334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 10646
I20260927 02:53:29.411350 11081 raft_consensus.cc:2258] T 862a1b32997449d8ac7056a8d004f58d P 7cdae604c02f4b0e83bce5b76c4c9e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:29.411478 11080 tablet_replica.cc:330] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: stopping tablet replica
I20260927 02:53:29.411559 11080 raft_consensus.cc:2229] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:53:29.411638 11080 raft_consensus.cc:2258] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:29.412002 11080 ts_tablet_manager.cc:1905] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:29.413544 11082 tablet_replica.cc:330] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: stopping tablet replica
I20260927 02:53:29.413574 11080 ts_tablet_manager.cc:1918] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:29.413641 11082 raft_consensus.cc:2229] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:53:29.413728 11082 raft_consensus.cc:2258] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:53:29.413738 11080 log.cc:1196] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/862a1b32997449d8ac7056a8d004f58d
I20260927 02:53:29.414089 11080 ts_tablet_manager.cc:1939] T 862a1b32997449d8ac7056a8d004f58d P 2d89980b63ca42b3acffb748b2340a1e: Deleting consensus metadata
I20260927 02:53:29.414146 11082 ts_tablet_manager.cc:1905] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:53:29.414556 10569 catalog_manager.cc:4841] TS 2d89980b63ca42b3acffb748b2340a1e (127.0.98.2:41831): tablet 862a1b32997449d8ac7056a8d004f58d (table test-table [id=851bcbd2da2543fc89f84d9c57accfb3]) successfully deleted
I20260927 02:53:29.415774 11082 ts_tablet_manager.cc:1918] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:53:29.415854 11082 log.cc:1196] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/862a1b32997449d8ac7056a8d004f58d
I20260927 02:53:29.416109 11082 ts_tablet_manager.cc:1939] T 862a1b32997449d8ac7056a8d004f58d P ba4abe054136406cb5d7fe5c69b7be15: Deleting consensus metadata
I20260927 02:53:29.416477 10569 catalog_manager.cc:4841] TS ba4abe054136406cb5d7fe5c69b7be15 (127.0.98.3:40035): tablet 862a1b32997449d8ac7056a8d004f58d (table test-table [id=851bcbd2da2543fc89f84d9c57accfb3]) successfully deleted
W20260927 02:53:29.416940 10569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34153 (error 108)
I20260927 02:53:29.417164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 10780
I20260927 02:53:29.417659 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:53:29.417250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34153 (local address 127.0.0.1:55720)
0927 02:53:29.417591 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 02:53:29.417628 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322}
W20260927 02:53:29.417840 10569 catalog_manager.cc:4584] TS 7cdae604c02f4b0e83bce5b76c4c9e6a (127.0.98.1:34153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 862a1b32997449d8ac7056a8d004f58d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34153: connect: Connection refused (error 111)
I20260927 02:53:29.421511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 10914
I20260927 02:53:29.425454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 10552
2026-09-27T02:53:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1772 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:53:29 dist-test-slave-mfx9 krb5kdc[11086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:53:29 dist-test-slave-mfx9 krb5kdc[11086](info): set up 2 sockets
Sep 27 02:53:29 dist-test-slave-mfx9 krb5kdc[11086](info): commencing operation
krb5kdc: starting...
W20260927 02:53:30.367606 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:53:30 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:53:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:53:30.516801 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskoFrgQ1/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:53:32.847 UTC [11116] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:53:32.847 UTC [11116] LOG: listening on IPv4 address "127.0.98.20", port 46383
2026-09-27 02:53:32.850 UTC [11116] LOG: listening on Unix socket "/tmp/.s.PGSQL.46383"
2026-09-27 02:53:32.878 UTC [11119] LOG: database system was shut down at 2026-09-27 02:53:31 UTC
2026-09-27 02:53:32.882 UTC [11116] LOG: database system is ready to accept connections
I20260927 02:53:33.747203 392 mini_postgres.cc:96] Postgres bound to 46383
2026-09-27 02:53:33.753 UTC [11127] FATAL: database "slave" does not exist
127.0.98.20:46383 - accepting connections
I20260927 02:53:33.753899 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:53:33.767683 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:53:34.213301 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:53:34.213368 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-admin
2026-09-27 02:53:34,475 [I] DB FLAVOR :POSTGRES
2026-09-27 02:53:34,476 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:53:34,476 [I] Checking connection..
2026-09-27 02:53:34,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:53:34,715 [I] Checking connection passed.
2026-09-27 02:53:34,715 [I] --------- Verifying version history table ---------
2026-09-27 02:53:34,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:53:34,946 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:53:34,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:53:35,180 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:53:35,180 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:53:35,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:53:35.400 UTC [11191] WARNING: there is no transaction in progress
2026-09-27 02:53:35,405 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:53:35,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:53:35,651 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:53:35,651 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:53:35,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:53:35,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:53:36,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mfx9.c.gcp-upstream.internal','N') ;"
2026-09-27 02:53:36,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:53:36,603 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:53:36,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:53:36,847 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:53:36,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:53:37,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:53:37,328 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:53:37,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:53:38.258 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.275 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.290 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.304 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.319 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.335 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.386 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.392 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.405 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.415 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.424 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.436 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.445 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.450 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.456 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:38.463 UTC [11299] WARNING: there is no transaction in progress
2026-09-27 02:53:39,041 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:53:39,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:53:39,253 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:53:39,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:53:39,490 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:53:39,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:53:39,737 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:53:39,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:53:39,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:53:40,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mfx9.c.gcp-upstream.internal';"
2026-09-27 02:53:40,437 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:53:40,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:53:40,683 [I] DB_PATCHES have already been applied
I20260927 02:53:40.688557 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskoFrgQ1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:53:40.688617 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:53:40.691256 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58981
Sep 27, 2026 2:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:53:41.666316 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.912s user 0.002s sys 0.005s
Sep 27, 2026 2:54:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:54:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:54:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:54:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:54:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:54:01.664443 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:54:01 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477641, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:54:01.738207 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:54:01.746351 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:54:02.522176 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:54:02.522256 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms
I20260927 02:54:02.522274 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:46383
2026-09-27 02:54:02,680 [I] DB FLAVOR :POSTGRES
2026-09-27 02:54:02,681 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:54:02,681 [I] Checking connection
2026-09-27 02:54:02,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:54:02,928 [I] connection success
2026-09-27 02:54:02,928 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:54:02,928 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:54:02,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:54:03,144 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:54:03,145 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:54:03,145 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:54:03,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:54:03,385 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:54:03.781682 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:54:03.963766 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:54:04.004019 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:54:03 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477641, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:04.391698 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:54:04.391810 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:54:04.391831 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:54:04.394793 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11479
I20260927 02:54:04.394910 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36173
Sep 27, 2026 2:54:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:54:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:54:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:54:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:54:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:54:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:54:06.356868 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.619s user 0.000s sys 0.141s
I20260927 02:54:06.356954 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:54:06.357031 392 mini_ranger_kms.cc:401] 127.0.98.20:36173/kms/v1/keys
Sep 27 02:54:06 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:54:06 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:54:07 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:54:09 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:54:09.634745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39481
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39481
--ranger_config_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36173/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:09.713207 11530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:09.713403 11530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:09.713423 11530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:09.714527 11530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:54:09.714566 11530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:09.714578 11530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:09.714589 11530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:54:09.714599 11530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:54:09.716125 11530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36173/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39481
--ranger_config_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:09.716324 11530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:09.716526 11530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:09.721529 11535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:09.721616 11530 server_base.cc:1034] running on GCE node
W20260927 02:54:09.721551 11538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:09.721736 11536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:09.722028 11530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:09.722365 11530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:09.723528 11530 hybrid_clock.cc:648] HybridClock initialized: now 1790477649723515 us; error 26 us; skew 500 ppm
Sep 27 02:54:09 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:09.726425 11530 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:54:09.726953 11530 webserver.cc:460] Webserver started at http://127.0.98.62:34869/ using document root <none> and password file <none>
I20260927 02:54:09.727203 11530 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:09.727258 11530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:09.727385 11530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:54:09 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:09.777261 11530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "72b2171a9f3047b2ada2650a055975d8"
format_stamp: "Formatted at 2026-09-27 02:54:09 on dist-test-slave-mfx9"
server_key: "77bc2f0573a3bbc4a47fffedd5c4c45c"
server_key_iv: "ca004a08766918d96edae910c3d18cbc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:09.777732 11530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "72b2171a9f3047b2ada2650a055975d8"
format_stamp: "Formatted at 2026-09-27 02:54:09 on dist-test-slave-mfx9"
server_key: "77bc2f0573a3bbc4a47fffedd5c4c45c"
server_key_iv: "ca004a08766918d96edae910c3d18cbc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:09.779593 11530 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 02:54:09.793597 11545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:09.793849 11530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:54:09.793965 11530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "72b2171a9f3047b2ada2650a055975d8"
format_stamp: "Formatted at 2026-09-27 02:54:09 on dist-test-slave-mfx9"
server_key: "77bc2f0573a3bbc4a47fffedd5c4c45c"
server_key_iv: "ca004a08766918d96edae910c3d18cbc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:09.794071 11530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:09.807219 11530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:09.809007 11530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:09.809129 11530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:09.813421 11530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39481
I20260927 02:54:09.813485 11597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39481 every 8 connection(s)
I20260927 02:54:09.813818 11530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:54:09.814606 11598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:09.817868 11598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8: Bootstrap starting.
I20260927 02:54:09.818614 11598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:09.818948 11598 log.cc:824] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:09.819693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 11530
I20260927 02:54:09.819759 11598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8: No bootstrap required, opened a new log
I20260927 02:54:09.819779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:54:09.819944 392 external_mini_cluster.cc:1428] Setting key 5d96052f598991ee8e55d5c7ffeeee76
I20260927 02:54:09.822829 11598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b2171a9f3047b2ada2650a055975d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } }
I20260927 02:54:09.823015 11598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:09.823052 11598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b2171a9f3047b2ada2650a055975d8, State: Initialized, Role: FOLLOWER
I20260927 02:54:09.823217 11598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b2171a9f3047b2ada2650a055975d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } }
I20260927 02:54:09.823326 11598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:54:09.823378 11598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:54:09.823438 11598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:09.824163 11598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b2171a9f3047b2ada2650a055975d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } }
I20260927 02:54:09.824286 11598 leader_election.cc:302] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72b2171a9f3047b2ada2650a055975d8; no voters:
I20260927 02:54:09.824461 11598 leader_election.cc:288] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:54:09.824851 11598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 02:54:09 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:09.824498 11603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:54:09.826871 11603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [term 1 LEADER]: Becoming Leader. State: Replica: 72b2171a9f3047b2ada2650a055975d8, State: Running, Role: LEADER
I20260927 02:54:09.827054 11603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b2171a9f3047b2ada2650a055975d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } }
I20260927 02:54:09.827405 11603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b2171a9f3047b2ada2650a055975d8. Latest consensus state: current_term: 1 leader_uuid: "72b2171a9f3047b2ada2650a055975d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b2171a9f3047b2ada2650a055975d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } }
I20260927 02:54:09.827486 11603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:09.828104 11598 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:54:09.830464 11615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72b2171a9f3047b2ada2650a055975d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b2171a9f3047b2ada2650a055975d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } }
I20260927 02:54:09.830746 11615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:09.831115 11617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:54:09.832229 11617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:54:09.832633 11602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:09.820449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.0.98.62:39481)
0927 02:54:09.820603 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:09.820613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:09.820641 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:54:09.821199 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:09.821206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:09.821253 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:09.821376 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:09.821949 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:09.823201 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:09.823991 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:09.824524 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:09.824626 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:09.826965 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:09.826969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:09.826969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:09.826982 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:09.830125 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:09.830554 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:09.830561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:09.830561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:09.830733 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:09.830963 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:09.830964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:09.830964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:09.831608 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:09.831864 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:09.831937 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 02:54:09.832380 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260927 02:54:09.835551 11617 catalog_manager.cc:1347] Generated new cluster ID: 8cb6188717264303a50b00a2d6b5b292
I20260927 02:54:09.835601 11617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:54:09.848124 11617 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:54:09.849220 11617 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:54:09.856868 11617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8: Generated new TSK 0
I20260927 02:54:09.857132 11617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:54:10.594655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36173/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39481
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:10.712672 11643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:10.712936 11643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:10.712971 11643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:10.714605 11643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:10.714664 11643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:10.714702 11643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:54:10.716751 11643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36173/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:10.716970 11643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:10.717211 11643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:10.722283 11648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:10.722398 11651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:10.722280 11649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:10.722447 11643 server_base.cc:1034] running on GCE node
I20260927 02:54:10.722682 11643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:10.722970 11643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:10.724149 11643 hybrid_clock.cc:648] HybridClock initialized: now 1790477650724132 us; error 33 us; skew 500 ppm
Sep 27 02:54:10 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:10.727159 11643 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:54:10.727550 11643 webserver.cc:460] Webserver started at http://127.0.98.1:41949/ using document root <none> and password file <none>
I20260927 02:54:10.727749 11643 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:10.727797 11643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:10.727916 11643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:54:10 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:10.740298 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0"
format_stamp: "Formatted at 2026-09-27 02:54:10 on dist-test-slave-mfx9"
server_key: "93c8be2b2b254835a8010d22116b2d1d"
server_key_iv: "27d000706c48688c45ec45e7548e0382"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:10.740732 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0"
format_stamp: "Formatted at 2026-09-27 02:54:10 on dist-test-slave-mfx9"
server_key: "93c8be2b2b254835a8010d22116b2d1d"
server_key_iv: "27d000706c48688c45ec45e7548e0382"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:10.742383 11643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:54:10.766932 11660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:10.767169 11643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260927 02:54:10.767269 11643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0"
format_stamp: "Formatted at 2026-09-27 02:54:10 on dist-test-slave-mfx9"
server_key: "93c8be2b2b254835a8010d22116b2d1d"
server_key_iv: "27d000706c48688c45ec45e7548e0382"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:10.767365 11643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:10.785071 11643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:10.787525 11643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:10.787727 11643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:10.788022 11643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:10.788439 11643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:10.788537 11643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:10.788628 11643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:10.788694 11643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:10.796047 11643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38017
I20260927 02:54:10.796499 11643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:54:10.797736 11773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38017 every 8 connection(s)
Sep 27 02:54:10 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:10.803444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 11643
I20260927 02:54:10.803550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:54:10.803723 392 external_mini_cluster.cc:1428] Setting key b9e29401010f621f822b27083b410737
I20260927 02:54:10.806993 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:10.798403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39481 (local address 127.0.98.1:57769)
0927 02:54:10.798555 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:10.798598 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:10.799111 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:10.799357 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:10.799361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:10.799531 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:10.799777 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:10.799782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:10.800451 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:10.800452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:10.801249 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:10.801252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:10.801593 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:10.801594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:10.801708 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:10.802262 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:10.802270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:10.803464 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:10.805436 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:10.805443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:10.805445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:10.805706 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:10.805909 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:10.805909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:10.805910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:10.806042 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:10.806306 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:10.806312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:10.806490 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 02:54:10.806575 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:54:10.806736 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260927 02:54:10.807950 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:10.798460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57769 (local address 127.0.98.62:39481)
0927 02:54:10.798706 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:10.798708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:10.798718 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:54:10.799208 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:10.799210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:10.799224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:10.799288 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:10.799866 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:10.800373 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:10.801326 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:10.801530 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:10.801582 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:10.803585 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:10.803587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:10.803587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:10.803594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:10.805353 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:10.805780 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:10.805781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:10.805781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:10.805844 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:10.806092 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:10.806093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:10.806093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:10.806232 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:10.807734 (+ 1502us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:10.807779 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 02:54:10.807862 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:54:10.822685 11774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39481
I20260927 02:54:10.822948 11774 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:10.823252 11774 heartbeater.cc:511] Master 127.0.98.62:39481 requested a full tablet report, sending...
I20260927 02:54:10.823961 11562 ts_manager.cc:194] Registered new tserver with Master: f1fe4d5d6f764fec8b3fdfe2e0217af0 (127.0.98.1:38017)
I20260927 02:54:10.824895 11562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:54:10.833549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36173/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39481
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:10.929922 11780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:10.930128 11780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:10.930158 11780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:10.931901 11780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:10.932010 11780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:10.932090 11780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:54:10.933703 11780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36173/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:10.933938 11780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:10.934161 11780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:10.939780 11787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:10.939781 11788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:10.940167 11790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:10.940474 11780 server_base.cc:1034] running on GCE node
I20260927 02:54:10.940680 11780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:10.940924 11780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:10.942077 11780 hybrid_clock.cc:648] HybridClock initialized: now 1790477650942060 us; error 33 us; skew 500 ppm
Sep 27 02:54:10 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:10.944548 11780 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:54:10.945009 11780 webserver.cc:460] Webserver started at http://127.0.98.2:37101/ using document root <none> and password file <none>
I20260927 02:54:10.945243 11780 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:10.945291 11780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:10.945420 11780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:54:10 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:54:10 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:10.957681 11780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "f396449affb641d5927345fc51068a43"
format_stamp: "Formatted at 2026-09-27 02:54:10 on dist-test-slave-mfx9"
server_key: "f4b6ced1d40d13f0b7b990152f5f0973"
server_key_iv: "67e9e5dc4cd922edb8962e4628dd9a58"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:10.959261 11780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f396449affb641d5927345fc51068a43"
format_stamp: "Formatted at 2026-09-27 02:54:10 on dist-test-slave-mfx9"
server_key: "f4b6ced1d40d13f0b7b990152f5f0973"
server_key_iv: "67e9e5dc4cd922edb8962e4628dd9a58"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:10.969338 11780 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260927 02:54:10.987583 11797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:10.987847 11780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 02:54:10.987949 11780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "f396449affb641d5927345fc51068a43"
format_stamp: "Formatted at 2026-09-27 02:54:10 on dist-test-slave-mfx9"
server_key: "f4b6ced1d40d13f0b7b990152f5f0973"
server_key_iv: "67e9e5dc4cd922edb8962e4628dd9a58"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:10.988046 11780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:11.001762 11780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:11.003762 11780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:11.003891 11780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:11.004089 11780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:11.004498 11780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:11.004586 11780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:11.004667 11780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:11.004720 11780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:11.011583 11780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44733
I20260927 02:54:11.011953 11780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:54:11.012420 11910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44733 every 8 connection(s)
I20260927 02:54:11.019702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 11780
I20260927 02:54:11.019807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:54:11.019960 392 external_mini_cluster.cc:1428] Setting key de9ce4fbfe2739da9d93ba3f05752359
Sep 27 02:54:11 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:11.029094 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.012631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41229 (local address 127.0.98.62:39481)
0927 02:54:11.012692 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.012694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.018854 (+ 6160us) server_negotiation.cc:407] Connection header received
0927 02:54:11.019466 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.019467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.019481 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.019546 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:11.020748 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.021447 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.022953 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.023226 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.023261 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.025682 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:11.025683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:11.025684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:11.025691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:11.027516 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:11.028011 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:11.028012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:11.028013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:11.028072 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:11.028345 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:11.028346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:11.028346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:11.028482 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:11.028521 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.028899 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.028999 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:54:11.029837 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.012639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39481 (local address 127.0.98.2:41229)
0927 02:54:11.018682 (+ 6043us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.018734 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:11.019352 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:11.019682 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:11.019686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:11.019925 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:11.020647 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.020655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.021557 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.021558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.022855 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.022858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.023334 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.023334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.023437 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.023954 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:11.023964 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:11.025563 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:11.027615 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:11.027622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:11.027624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:11.027919 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:11.028135 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:11.028136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:11.028136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:11.028268 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:11.028562 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:11.028569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:11.028746 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 02:54:11.029429 (+ 683us) client_negotiation.cc:240] Negotiation successful
0927 02:54:11.029595 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5997,"thread_start_us":76,"threads_started":1}
I20260927 02:54:11.030515 11911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39481
I20260927 02:54:11.030609 11911 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:11.030833 11911 heartbeater.cc:511] Master 127.0.98.62:39481 requested a full tablet report, sending...
I20260927 02:54:11.031261 11562 ts_manager.cc:194] Registered new tserver with Master: f396449affb641d5927345fc51068a43 (127.0.98.2:44733)
I20260927 02:54:11.031742 11562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41229
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:54:11.055887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36173/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39481
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:11.180897 11917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:11.188793 11917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:11.188913 11917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:11.191185 11917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:11.191303 11917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:11.191392 11917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:54:11.193912 11917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36173/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:11.194378 11917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:11.194706 11917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:11.206489 11925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:11.206512 11923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:11.206947 11922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:11.208030 11917 server_base.cc:1034] running on GCE node
I20260927 02:54:11.208233 11917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:11.208475 11917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:11.215819 11917 hybrid_clock.cc:648] HybridClock initialized: now 1790477651215792 us; error 35 us; skew 500 ppm
Sep 27 02:54:11 dist-test-slave-mfx9 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:11.219007 11917 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:54:11.219491 11917 webserver.cc:460] Webserver started at http://127.0.98.3:37797/ using document root <none> and password file <none>
I20260927 02:54:11.219745 11917 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:11.219791 11917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:11.219903 11917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:54:11 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:11.235981 11917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb02e70742eb40fd8784b4fcaf9770c1"
format_stamp: "Formatted at 2026-09-27 02:54:11 on dist-test-slave-mfx9"
server_key: "72463de8b8013ba345c4373359fdf711"
server_key_iv: "3af3904852902081dec5a4e2b75ab4dc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:11.236428 11917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb02e70742eb40fd8784b4fcaf9770c1"
format_stamp: "Formatted at 2026-09-27 02:54:11 on dist-test-slave-mfx9"
server_key: "72463de8b8013ba345c4373359fdf711"
server_key_iv: "3af3904852902081dec5a4e2b75ab4dc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:11.240172 11917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 02:54:11.255949 11932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:11.256109 11917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:54:11.256198 11917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "cb02e70742eb40fd8784b4fcaf9770c1"
format_stamp: "Formatted at 2026-09-27 02:54:11 on dist-test-slave-mfx9"
server_key: "72463de8b8013ba345c4373359fdf711"
server_key_iv: "3af3904852902081dec5a4e2b75ab4dc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:11.256286 11917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:11.289885 11917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:11.292325 11917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:11.292454 11917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:11.292680 11917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:11.293090 11917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:11.293181 11917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:11.293258 11917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:11.293315 11917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:11.303179 11917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45851
I20260927 02:54:11.303706 11917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:54:11.306072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 11917
I20260927 02:54:11.306174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:54:11.306362 392 external_mini_cluster.cc:1428] Setting key 586c17c2922b11896fee1d1973d7dd3b
I20260927 02:54:11.311209 12045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45851 every 8 connection(s)
Sep 27 02:54:11 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:11.321702 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.304637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34589 (local address 127.0.98.62:39481)
0927 02:54:11.304688 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.304690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.311576 (+ 6886us) server_negotiation.cc:407] Connection header received
0927 02:54:11.312189 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.312191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.312204 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.312280 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:11.312952 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.313738 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.315149 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.315466 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.315539 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.318405 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:11.318406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:11.318406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:11.318413 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:11.320129 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:11.320644 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:11.320645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:11.320645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:11.320708 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:11.320990 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:11.320990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:11.320991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:11.321137 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:11.321203 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.321528 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.321610 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:54:11.321885 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.304609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39481 (local address 127.0.98.3:34589)
0927 02:54:11.311429 (+ 6820us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.311478 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:11.312079 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:11.312359 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:11.312364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:11.312588 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:11.312865 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.312872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.313835 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.313836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.315051 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.315054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.315566 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.315568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.315774 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.316422 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:11.316434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:11.318331 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:11.320243 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:11.320250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:11.320253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:11.320556 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:11.320772 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:11.320772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:11.320773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:11.320922 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:11.321202 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:11.321206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:11.321378 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 02:54:11.321497 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 02:54:11.321659 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6769,"thread_start_us":75,"threads_started":1}
I20260927 02:54:11.322517 12046 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39481
I20260927 02:54:11.322628 12046 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:11.322847 12046 heartbeater.cc:511] Master 127.0.98.62:39481 requested a full tablet report, sending...
I20260927 02:54:11.323264 11562 ts_manager.cc:194] Registered new tserver with Master: cb02e70742eb40fd8784b4fcaf9770c1 (127.0.98.3:45851)
I20260927 02:54:11.323652 11562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34589
I20260927 02:54:11.336678 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:54:11.346395 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.338365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.0.98.62:39481)
0927 02:54:11.338406 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.338408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.338691 (+ 283us) server_negotiation.cc:407] Connection header received
0927 02:54:11.338814 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.338815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.338825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.338854 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:11.339704 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.340534 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.341756 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.342037 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.342082 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.343594 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:11.343595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:11.343596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:11.343602 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:11.345192 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:11.345569 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:11.345570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:11.345571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:11.345652 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:11.345860 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:11.345861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:11.345861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:11.345991 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:11.346096 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.346208 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.346293 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:54:11.348114 11562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:54:11.350298 11562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:54:11.368008 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.354224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38017 (local address 127.0.0.1:42926)
0927 02:54:11.355778 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.355787 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:11.355820 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:11.356462 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:11.356465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:11.356467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:11.356530 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.356532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.364090 (+ 7558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.364091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.365149 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.365151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.367699 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.367700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.367778 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.367792 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:54:11.367866 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:54:11.367917 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1541,"thread_start_us":61,"threads_started":1}
I20260927 02:54:11.370080 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.353509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44733 (local address 127.0.0.1:35682)
0927 02:54:11.355563 (+ 2054us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.355616 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:11.355677 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:11.357386 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:11.357387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:11.357390 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:11.357448 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.357450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.358707 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.358708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.365493 (+ 6785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.365495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.369395 (+ 3900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.369396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.369469 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.369474 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:54:11.369532 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:54:11.369573 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023,"thread_start_us":547,"threads_started":1}
I20260927 02:54:11.370703 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.354385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45851 (local address 127.0.0.1:44652)
0927 02:54:11.359705 (+ 5320us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.359716 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:11.359763 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:11.360416 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:11.360417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:11.360420 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:11.360460 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.360461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.362442 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.362443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.363613 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.363615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.370452 (+ 6837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.370453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.370518 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.370522 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:54:11.370575 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:54:11.370629 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5302,"thread_start_us":64,"threads_started":1}
I20260927 02:54:11.371065 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.355113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.98.3:45851)
0927 02:54:11.356417 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.356420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.359824 (+ 3404us) server_negotiation.cc:407] Connection header received
0927 02:54:11.359869 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.359872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.359888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.359956 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:11.360546 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.361629 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.365865 (+ 4236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.366577 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.366664 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.366740 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.370936 (+ 4196us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.370982 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":70,"threads_started":1}
I20260927 02:54:11.371403 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.354980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.0.98.1:38017)
0927 02:54:11.356181 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.356185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.356194 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:54:11.356224 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.356226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.356241 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.356307 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:11.356627 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.364004 (+ 7377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.366806 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.367516 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.367604 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.367661 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.371286 (+ 3625us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.371335 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":54,"threads_started":1}
I20260927 02:54:11.371724 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.354814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.0.98.2:44733)
0927 02:54:11.355932 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.355937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.355947 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:54:11.355986 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.355988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.356004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.356073 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:11.357528 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.358606 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.368386 (+ 9780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.369167 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.369249 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.369356 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.371581 (+ 2225us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.371639 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1081,"thread_start_us":81,"threads_started":1}
I20260927 02:54:11.372627 11980 tablet_service.cc:1467] Processing CreateTablet for tablet cc749eb925ff42f4a79e07bae5475ffa (DEFAULT_TABLE table=test-table [id=7f2cd3d7c0f5454ea00c149d556f5803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:54:11.372999 11980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc749eb925ff42f4a79e07bae5475ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:11.377611 11708 tablet_service.cc:1467] Processing CreateTablet for tablet cc749eb925ff42f4a79e07bae5475ffa (DEFAULT_TABLE table=test-table [id=7f2cd3d7c0f5454ea00c149d556f5803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:54:11.377946 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc749eb925ff42f4a79e07bae5475ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:11.382804 11828 tablet_service.cc:1467] Processing CreateTablet for tablet cc749eb925ff42f4a79e07bae5475ffa (DEFAULT_TABLE table=test-table [id=7f2cd3d7c0f5454ea00c149d556f5803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:54:11.383141 11828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc749eb925ff42f4a79e07bae5475ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:11.386893 12066 tablet_bootstrap.cc:492] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0: Bootstrap starting.
I20260927 02:54:11.387668 12066 tablet_bootstrap.cc:654] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:11.388036 12066 log.cc:824] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:11.388444 12067 tablet_bootstrap.cc:492] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43: Bootstrap starting.
I20260927 02:54:11.390024 12067 tablet_bootstrap.cc:654] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:11.390473 12065 tablet_bootstrap.cc:492] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: Bootstrap starting.
I20260927 02:54:11.393360 12065 tablet_bootstrap.cc:654] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:11.393715 12067 log.cc:824] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:11.393715 12065 log.cc:824] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:11.394447 12067 tablet_bootstrap.cc:492] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43: No bootstrap required, opened a new log
I20260927 02:54:11.394510 12067 ts_tablet_manager.cc:1397] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 02:54:11.395918 12067 raft_consensus.cc:348] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.396052 12067 raft_consensus.cc:374] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:11.396081 12067 raft_consensus.cc:729] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f396449affb641d5927345fc51068a43, State: Initialized, Role: FOLLOWER
I20260927 02:54:11.396204 12067 consensus_queue.cc:260] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.396457 12067 ts_tablet_manager.cc:1428] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:54:11.396477 11911 heartbeater.cc:503] Master 127.0.98.62:39481 was elected leader, sending a full tablet report...
I20260927 02:54:11.396870 12066 tablet_bootstrap.cc:492] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0: No bootstrap required, opened a new log
I20260927 02:54:11.396960 12066 ts_tablet_manager.cc:1397] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 02:54:11.399089 12066 raft_consensus.cc:348] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.399312 12066 raft_consensus.cc:374] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:11.399385 12066 raft_consensus.cc:729] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fe4d5d6f764fec8b3fdfe2e0217af0, State: Initialized, Role: FOLLOWER
I20260927 02:54:11.399552 12066 consensus_queue.cc:260] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.399819 12066 ts_tablet_manager.cc:1428] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:54:11.399950 11774 heartbeater.cc:503] Master 127.0.98.62:39481 was elected leader, sending a full tablet report...
I20260927 02:54:11.401204 12065 tablet_bootstrap.cc:492] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: No bootstrap required, opened a new log
I20260927 02:54:11.401286 12065 ts_tablet_manager.cc:1397] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260927 02:54:11.403632 12065 raft_consensus.cc:348] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.403779 12065 raft_consensus.cc:374] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:11.403806 12065 raft_consensus.cc:729] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb02e70742eb40fd8784b4fcaf9770c1, State: Initialized, Role: FOLLOWER
I20260927 02:54:11.403898 12065 consensus_queue.cc:260] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.404115 12065 ts_tablet_manager.cc:1428] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 02:54:11.404342 12046 heartbeater.cc:503] Master 127.0.98.62:39481 was elected leader, sending a full tablet report...
Sep 27 02:54:11 dist-test-slave-mfx9 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 02:54:11.513173 11912 tablet.cc:2369] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:54:11.542644 12071 raft_consensus.cc:482] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:54:11.542807 12071 raft_consensus.cc:504] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.543220 12071 leader_election.cc:288] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb02e70742eb40fd8784b4fcaf9770c1 (127.0.98.3:45851), f1fe4d5d6f764fec8b3fdfe2e0217af0 (127.0.98.1:38017)
I20260927 02:54:11.547322 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.543768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56781 (local address 127.0.98.1:38017)
0927 02:54:11.543827 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.543830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.544051 (+ 221us) server_negotiation.cc:407] Connection header received
0927 02:54:11.544142 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.544143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.544154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.544178 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:11.544358 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.545060 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.546394 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.546978 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.547020 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.547030 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.547218 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.547255 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:54:11.547712 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.543466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38017 (local address 127.0.98.2:56781)
0927 02:54:11.543999 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.544008 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:11.544077 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:11.544236 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:11.544237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:11.544238 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:11.544295 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.544297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.545154 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.545155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.546300 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.546304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.547083 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.547083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.547149 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.547153 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:54:11.547590 (+ 437us) client_negotiation.cc:240] Negotiation successful
0927 02:54:11.547642 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":66,"threads_started":1}
I20260927 02:54:11.548134 11720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc749eb925ff42f4a79e07bae5475ffa" candidate_uuid: "f396449affb641d5927345fc51068a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" is_pre_election: true
I20260927 02:54:11.548296 11720 raft_consensus.cc:2454] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f396449affb641d5927345fc51068a43 in term 0.
I20260927 02:54:11.548640 11799 leader_election.cc:302] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1fe4d5d6f764fec8b3fdfe2e0217af0, f396449affb641d5927345fc51068a43; no voters:
W20260927 02:54:11.549190 11775 tablet.cc:2369] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:54:11.549404 12071 raft_consensus.cc:2781] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:54:11.549456 12071 raft_consensus.cc:482] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:54:11.549477 12071 raft_consensus.cc:3037] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:11.550258 12071 raft_consensus.cc:504] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.550375 12071 leader_election.cc:288] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [CANDIDATE]: Term 1 election: Requested vote from peers cb02e70742eb40fd8784b4fcaf9770c1 (127.0.98.3:45851), f1fe4d5d6f764fec8b3fdfe2e0217af0 (127.0.98.1:38017)
I20260927 02:54:11.550653 11720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc749eb925ff42f4a79e07bae5475ffa" candidate_uuid: "f396449affb641d5927345fc51068a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0"
I20260927 02:54:11.550721 11720 raft_consensus.cc:3037] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:11.551509 11720 raft_consensus.cc:2454] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f396449affb641d5927345fc51068a43 in term 1.
I20260927 02:54:11.551707 11799 leader_election.cc:302] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1fe4d5d6f764fec8b3fdfe2e0217af0, f396449affb641d5927345fc51068a43; no voters:
I20260927 02:54:11.551805 12071 raft_consensus.cc:2781] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:54:11.551923 12071 raft_consensus.cc:686] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [term 1 LEADER]: Becoming Leader. State: Replica: f396449affb641d5927345fc51068a43, State: Running, Role: LEADER
I20260927 02:54:11.551997 12071 consensus_queue.cc:237] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } }
I20260927 02:54:11.552779 11562 catalog_manager.cc:5495] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to f396449affb641d5927345fc51068a43 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f396449affb641d5927345fc51068a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396449affb641d5927345fc51068a43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 } health_report { overall_health: UNKNOWN } } }
I20260927 02:54:11.555246 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.543969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43149 (local address 127.0.98.3:45851)
0927 02:54:11.544010 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.544012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.548061 (+ 4049us) server_negotiation.cc:407] Connection header received
0927 02:54:11.548152 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.548154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.548166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.548190 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:11.548322 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.549029 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.554379 (+ 5350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.554918 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.554959 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.554968 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.555143 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.555178 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:54:11.555557 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.543895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45851 (local address 127.0.98.2:43149)
0927 02:54:11.548016 (+ 4121us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.548025 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:11.548086 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:11.548239 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:11.548240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:11.548241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:11.548266 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.548267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.553262 (+ 4995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.553263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.554252 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:11.554254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.555016 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.555017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:11.555073 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.555075 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:54:11.555458 (+ 383us) client_negotiation.cc:240] Negotiation successful
0927 02:54:11.555499 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4111,"thread_start_us":56,"threads_started":1}
I20260927 02:54:11.555984 11981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc749eb925ff42f4a79e07bae5475ffa" candidate_uuid: "f396449affb641d5927345fc51068a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" is_pre_election: true
I20260927 02:54:11.556147 11981 raft_consensus.cc:2454] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f396449affb641d5927345fc51068a43 in term 0.
I20260927 02:54:11.556262 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc749eb925ff42f4a79e07bae5475ffa" candidate_uuid: "f396449affb641d5927345fc51068a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb02e70742eb40fd8784b4fcaf9770c1"
I20260927 02:54:11.556313 12000 raft_consensus.cc:3037] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:11.557127 12000 raft_consensus.cc:2454] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f396449affb641d5927345fc51068a43 in term 1.
W20260927 02:54:11.561596 12047 tablet.cc:2369] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:54:11.583005 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.579841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.0.98.2:44733)
0927 02:54:11.579906 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:11.579909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:11.579921 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:54:11.579962 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:11.579963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:11.579985 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:11.580078 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:54:11.580218 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.580922 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.581980 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:11.582262 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:11.582314 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:11.582473 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:54:11.582590 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:54:11.582683 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:11.582919 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 02:54:11.582937 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:54:11.585093 11720 raft_consensus.cc:1261] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 1 FOLLOWER]: Refusing update from remote peer f396449affb641d5927345fc51068a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:54:11.585469 12076 consensus_queue.cc:1035] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1fe4d5d6f764fec8b3fdfe2e0217af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:54:11.587462 12000 raft_consensus.cc:1261] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 1 FOLLOWER]: Refusing update from remote peer f396449affb641d5927345fc51068a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:54:11.587741 12076 consensus_queue.cc:1035] T cc749eb925ff42f4a79e07bae5475ffa P f396449affb641d5927345fc51068a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb02e70742eb40fd8784b4fcaf9770c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:54:11.589144 12079 mvcc.cc:205] Tried to move back new op lower bound from 7333796460889698304 to 7333796460757422080. Current Snapshot: MvccSnapshot[applied={T|T < 7333796460889698304}]
I20260927 02:54:11.589671 12080 mvcc.cc:205] Tried to move back new op lower bound from 7333796460889698304 to 7333796460757422080. Current Snapshot: MvccSnapshot[applied={T|T < 7333796460889698304}]
I20260927 02:54:11.600660 11562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:54:11.600764 11562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:54:11.601523 11562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72b2171a9f3047b2ada2650a055975d8: Sending DeleteTablet for 3 replicas of tablet cc749eb925ff42f4a79e07bae5475ffa
I20260927 02:54:11.601852 11980 tablet_service.cc:1514] Processing DeleteTablet for tablet cc749eb925ff42f4a79e07bae5475ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:54:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260927 02:54:11.601981 11828 tablet_service.cc:1514] Processing DeleteTablet for tablet cc749eb925ff42f4a79e07bae5475ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:54:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35682
I20260927 02:54:11.602128 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet cc749eb925ff42f4a79e07bae5475ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:54:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42926
I20260927 02:54:11.602308 12090 tablet_replica.cc:330] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0: stopping tablet replica
I20260927 02:54:11.602393 12090 raft_consensus.cc:2229] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:54:11.602480 12090 raft_consensus.cc:2258] T cc749eb925ff42f4a79e07bae5475ffa P f1fe4d5d6f764fec8b3fdfe2e0217af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:54:11.602602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 11643
W20260927 02:54:11.606647 11547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38017 (error 108)
I20260927 02:54:11.606961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 11780
I20260927 02:54:11.607160 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.607028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38017 (local address 127.0.0.1:42942)
0927 02:54:11.607072 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.607129 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:54:11.607484 11547 catalog_manager.cc:4584] TS f1fe4d5d6f764fec8b3fdfe2e0217af0 (127.0.98.1:38017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc749eb925ff42f4a79e07bae5475ffa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38017: connect: Connection refused (error 111)
I20260927 02:54:11.609442 12088 tablet_replica.cc:330] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: stopping tablet replica
I20260927 02:54:11.609545 12088 raft_consensus.cc:2229] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:54:11.609654 12088 raft_consensus.cc:2258] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:54:11.612331 12088 ts_tablet_manager.cc:1905] T cc749eb925ff42f4a79e07bae5475ffa P cb02e70742eb40fd8784b4fcaf9770c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:54:11.613771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 11917
I20260927 02:54:11.613807 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.613713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44733 (local address 127.0.0.1:35694)
0927 02:54:11.613756 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.613777 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:54:11.618788 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:11.618682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45851 (local address 127.0.0.1:44662)
0927 02:54:11.618718 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:11.618745 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 02:54:11.618909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 11530
I20260927 02:54:11.624639 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:54:11.958400 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:54:11.958490 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:54:12.292232 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:54:12.292 UTC [11116] LOG: received smart shutdown request
2026-09-27 02:54:12.295 UTC [11116] LOG: background worker "logical replication launcher" (PID 11125) exited with exit code 1
2026-09-27 02:54:12.295 UTC [11120] LOG: shutting down
2026-09-27 02:54:12.440 UTC [11116] LOG: database system is shut down
2026-09-27T02:54:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43095 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:54:12 dist-test-slave-mfx9 krb5kdc[12100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:54:12 dist-test-slave-mfx9 krb5kdc[12100](info): set up 2 sockets
Sep 27 02:54:12 dist-test-slave-mfx9 krb5kdc[12100](info): commencing operation
krb5kdc: starting...
W20260927 02:54:13.470397 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:54:13 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:54:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:54:13.615487 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskoFrgQ1/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:54:15.945 UTC [12130] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:54:15.945 UTC [12130] LOG: listening on IPv4 address "127.0.98.20", port 43055
2026-09-27 02:54:15.948 UTC [12130] LOG: listening on Unix socket "/tmp/.s.PGSQL.43055"
2026-09-27 02:54:15.967 UTC [12133] LOG: database system was shut down at 2026-09-27 02:54:14 UTC
2026-09-27 02:54:15.970 UTC [12130] LOG: database system is ready to accept connections
I20260927 02:54:16.846719 392 mini_postgres.cc:96] Postgres bound to 43055
2026-09-27 02:54:16.852 UTC [12141] FATAL: database "slave" does not exist
127.0.98.20:43055 - accepting connections
I20260927 02:54:16.852512 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:54:16.866081 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:54:17.305887 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:54:17.305955 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-admin
2026-09-27 02:54:17,564 [I] DB FLAVOR :POSTGRES
2026-09-27 02:54:17,564 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:54:17,564 [I] Checking connection..
2026-09-27 02:54:17,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:54:17,812 [I] Checking connection passed.
2026-09-27 02:54:17,813 [I] --------- Verifying version history table ---------
2026-09-27 02:54:17,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:54:18,052 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:54:18,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:54:18,288 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:54:18,288 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:54:18,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:54:18.539 UTC [12205] WARNING: there is no transaction in progress
2026-09-27 02:54:18,544 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:54:18,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:54:18,773 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:54:18,774 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:54:18,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:54:18,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:54:19,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mfx9.c.gcp-upstream.internal','N') ;"
2026-09-27 02:54:19,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:54:19,726 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:54:19,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:54:19,947 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:54:19,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:54:20,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:54:20,408 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:54:20,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:54:21.291 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.306 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.323 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.338 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.353 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.370 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.422 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.428 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.441 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.450 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.460 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.472 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.481 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.487 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.493 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:21.499 UTC [12313] WARNING: there is no transaction in progress
2026-09-27 02:54:22,077 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:54:22,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:54:22,312 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:54:22,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:54:22,565 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:54:22,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:54:22,798 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:54:22,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:54:23,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:54:23,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mfx9.c.gcp-upstream.internal';"
2026-09-27 02:54:23,485 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:54:23,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:54:23,738 [I] DB_PATCHES have already been applied
I20260927 02:54:23.743826 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskoFrgQ1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:54:23.743884 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:54:23.746544 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36983
Sep 27, 2026 2:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:54:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:54:24.741168 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.889s user 0.000s sys 0.007s
Sep 27, 2026 2:54:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:54:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:54:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:54:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:54:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:54:45.902125 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:54:45 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477685, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:54:45.978740 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:54:45.986879 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:54:46.808187 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:54:46.808266 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms
I20260927 02:54:46.808284 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43055
2026-09-27 02:54:46,949 [I] DB FLAVOR :POSTGRES
2026-09-27 02:54:46,949 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:54:46,949 [I] Checking connection
2026-09-27 02:54:46,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:54:47,179 [I] connection success
2026-09-27 02:54:47,179 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:54:47,179 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:54:47,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:54:47,420 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:54:47,421 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:54:47,421 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:54:47,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:54:47,656 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:54:48.104692 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:54:48.286458 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:54:48.325176 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:54:48 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477685, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:48.669776 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:54:48.669880 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskoFrgQ1/build/release/bin/postgresql.jar:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskoFrgQ1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:54:48.669896 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:54:48.673254 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12493
I20260927 02:54:48.673310 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33409
Sep 27, 2026 2:54:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:54:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:54:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:54:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:54:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:54:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 02:54:50 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477690, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:50.722416 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.744s user 0.003s sys 0.120s
I20260927 02:54:50.722517 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:54:50.722564 392 mini_ranger_kms.cc:401] 127.0.98.20:33409/kms/v1/keys
Sep 27 02:54:51 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477690, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:54:51 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477690, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:54:53 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:54:53.591614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42911
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42911
--ranger_config_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33409/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:53.666734 12544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:53.666962 12544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:53.666988 12544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:53.668058 12544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:54:53.668095 12544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:53.668113 12544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:53.668128 12544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:54:53.668143 12544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:54:53.669629 12544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33409/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42911
--ranger_config_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:53.669839 12544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:53.670023 12544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:53.674428 12549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:53.674438 12552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:53.674487 12544 server_base.cc:1034] running on GCE node
W20260927 02:54:53.674439 12550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:53.674911 12544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:53.675206 12544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:53.676353 12544 hybrid_clock.cc:648] HybridClock initialized: now 1790477693676341 us; error 25 us; skew 500 ppm
Sep 27 02:54:53 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:53.683228 12544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:54:53.683629 12544 webserver.cc:460] Webserver started at http://127.0.98.62:38169/ using document root <none> and password file <none>
I20260927 02:54:53.683821 12544 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:53.683861 12544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:53.683990 12544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:54:53 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:53.737912 12544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "c8adf49c4ee24d57bba2c7a896474931"
format_stamp: "Formatted at 2026-09-27 02:54:53 on dist-test-slave-mfx9"
server_key: "7662328441da52b09e385660c8fe8f60"
server_key_iv: "208948a792ff66c0febbfb3724d956bc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:53.738288 12544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8adf49c4ee24d57bba2c7a896474931"
format_stamp: "Formatted at 2026-09-27 02:54:53 on dist-test-slave-mfx9"
server_key: "7662328441da52b09e385660c8fe8f60"
server_key_iv: "208948a792ff66c0febbfb3724d956bc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:53.739715 12544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:54:53.750914 12559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:53.751299 12544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:54:53.751410 12544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "c8adf49c4ee24d57bba2c7a896474931"
format_stamp: "Formatted at 2026-09-27 02:54:53 on dist-test-slave-mfx9"
server_key: "7662328441da52b09e385660c8fe8f60"
server_key_iv: "208948a792ff66c0febbfb3724d956bc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:53.751505 12544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:53.808550 12544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:53.810385 12544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:53.810494 12544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:53.814811 12544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42911
I20260927 02:54:53.814878 12611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42911 every 8 connection(s)
I20260927 02:54:53.815202 12544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:54:53.815824 12612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:53.816474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 12544
I20260927 02:54:53.816556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:54:53.816727 392 external_mini_cluster.cc:1428] Setting key 5c4818ae6bf0789ab4127c4ae2d4a54a
I20260927 02:54:53.818692 12612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Bootstrap starting.
I20260927 02:54:53.819350 12612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:53.819626 12612 log.cc:824] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:53.820390 12612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: No bootstrap required, opened a new log
I20260927 02:54:53.822376 12612 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } }
I20260927 02:54:53.822536 12612 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:53.822608 12612 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8adf49c4ee24d57bba2c7a896474931, State: Initialized, Role: FOLLOWER
I20260927 02:54:53.822745 12612 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } }
I20260927 02:54:53.822850 12612 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:54:53.822911 12612 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:54:53.822983 12612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:53.823603 12612 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } }
I20260927 02:54:53.823704 12612 leader_election.cc:302] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8adf49c4ee24d57bba2c7a896474931; no voters:
I20260927 02:54:53.823873 12612 leader_election.cc:288] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:54:53.823910 12617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:54:53.824074 12617 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 1 LEADER]: Becoming Leader. State: Replica: c8adf49c4ee24d57bba2c7a896474931, State: Running, Role: LEADER
I20260927 02:54:53.824190 12617 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } }
I20260927 02:54:53.824223 12612 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 02:54:53 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:53.825857 12618 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8adf49c4ee24d57bba2c7a896474931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } } }
I20260927 02:54:53.825934 12618 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:53.826174 12612 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:54:53.826145 12619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8adf49c4ee24d57bba2c7a896474931. Latest consensus state: current_term: 1 leader_uuid: "c8adf49c4ee24d57bba2c7a896474931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } } }
I20260927 02:54:53.826228 12619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:53.826637 12631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:54:53.828716 12631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:54:53.830544 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:53.817225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.0.98.62:42911)
0927 02:54:53.817403 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:53.817412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:53.817516 (+ 104us) server_negotiation.cc:407] Connection header received
0927 02:54:53.818000 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:53.818009 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:53.818080 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:53.818223 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:53.818999 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:53.820161 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:53.821238 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:53.821887 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:53.821994 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:53.824871 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:53.824875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:53.824876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:53.824887 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:53.826790 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:53.827914 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:53.827952 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:53.827952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:53.828407 (+ 455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:53.828653 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:53.828654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:53.828654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:53.829297 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:53.829596 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:53.829685 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:54:53.830202 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260927 02:54:53.835937 12631 catalog_manager.cc:1347] Generated new cluster ID: b730be613b904469a50af613b863e054
I20260927 02:54:53.835978 12631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:54:53.857240 12631 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:54:53.858018 12631 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:54:53.865548 12631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Generated new TSK 0
I20260927 02:54:53.865798 12631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:54:54.561152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42911
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:54.659292 12657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:54.659533 12657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:54.659564 12657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:54.661185 12657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:54.661237 12657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:54.661273 12657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:54:54.662853 12657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:54.663084 12657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:54.663362 12657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:54.670122 12665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:54.670112 12663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:54.670252 12662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:54.670388 12657 server_base.cc:1034] running on GCE node
I20260927 02:54:54.670562 12657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:54.670790 12657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:54.671939 12657 hybrid_clock.cc:648] HybridClock initialized: now 1790477694671923 us; error 31 us; skew 500 ppm
Sep 27 02:54:54 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:54.674357 12657 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:54:54.674793 12657 webserver.cc:460] Webserver started at http://127.0.98.1:37423/ using document root <none> and password file <none>
I20260927 02:54:54.674984 12657 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:54.675030 12657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:54.675144 12657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:54:54 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:54.688691 12657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "4680eafcf8d744d88e6b536d947accbc"
format_stamp: "Formatted at 2026-09-27 02:54:54 on dist-test-slave-mfx9"
server_key: "1402d98562d7a3e6721c966b61c70355"
server_key_iv: "5bcd2d27477a8cf5e578facab076c368"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:54.689105 12657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4680eafcf8d744d88e6b536d947accbc"
format_stamp: "Formatted at 2026-09-27 02:54:54 on dist-test-slave-mfx9"
server_key: "1402d98562d7a3e6721c966b61c70355"
server_key_iv: "5bcd2d27477a8cf5e578facab076c368"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:54.690748 12657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:54:54.702714 12673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:54.702914 12657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:54:54.703011 12657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "4680eafcf8d744d88e6b536d947accbc"
format_stamp: "Formatted at 2026-09-27 02:54:54 on dist-test-slave-mfx9"
server_key: "1402d98562d7a3e6721c966b61c70355"
server_key_iv: "5bcd2d27477a8cf5e578facab076c368"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:54.703096 12657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:54.725720 12657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:54.728128 12657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:54.728335 12657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:54.728616 12657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:54.728991 12657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:54.729074 12657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:54.729162 12657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:54.729226 12657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:54.735724 12657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46231
I20260927 02:54:54.736193 12657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:54:54.737313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 12657
I20260927 02:54:54.737391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:54:54.737552 392 external_mini_cluster.cc:1428] Setting key 3e28f3af48fd89cc5836bc414bed297f
I20260927 02:54:54.738852 12787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46231 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 02:54:54 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:54.755576 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:54.739479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40285 (local address 127.0.98.62:42911)
0927 02:54:54.743785 (+ 4306us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:54.743787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:54.743795 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:54:54.743819 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:54.743820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:54.743832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:54.743898 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:54.744708 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:54.745498 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:54.747007 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:54.747316 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:54.747383 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:54.752228 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:54.752230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:54.752231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:54.752238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:54.754008 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:54.754464 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:54.754465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:54.754465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:54.754525 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:54.754827 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:54.754828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:54.754829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:54.754981 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:54.755102 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:54.755385 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 02:54:54.755465 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4275,"thread_start_us":91,"threads_started":1}
I20260927 02:54:54.755718 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:54.739467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42911 (local address 127.0.98.1:40285)
0927 02:54:54.742914 (+ 3447us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:54.742966 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:54.743596 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:54.744069 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:54.744076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:54.744313 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:54.744595 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:54.744602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:54.745618 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:54.745619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:54.746915 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:54.746918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:54.747462 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:54.747463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:54.747610 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:54.748169 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:54.748178 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:54.752122 (+ 3944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:54.754107 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:54.754113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:54.754116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:54.754383 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:54.754587 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:54.754587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:54.754588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:54.754719 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:54.755058 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:54.755063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:54.755238 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 02:54:54.755342 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 02:54:54.755495 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3396,"thread_start_us":96,"threads_started":1}
I20260927 02:54:54.756349 12788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42911
I20260927 02:54:54.756451 12788 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:54.756690 12788 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:54.757248 12576 ts_manager.cc:194] Registered new tserver with Master: 4680eafcf8d744d88e6b536d947accbc (127.0.98.1:46231)
I20260927 02:54:54.758136 12576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:54:54.765784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42911
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:54.875839 12794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:54.876117 12794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:54.876156 12794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 27 02:54:54 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 02:54:54.877802 12794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:54.877864 12794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:54.877907 12794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:54:54.880071 12794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:54.880303 12794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:54.880611 12794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:54.887183 12802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:54.887612 12801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:54.888259 12804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:54.888695 12794 server_base.cc:1034] running on GCE node
I20260927 02:54:54.888898 12794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:54.889159 12794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:54.890334 12794 hybrid_clock.cc:648] HybridClock initialized: now 1790477694890312 us; error 35 us; skew 500 ppm
Sep 27 02:54:54 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:54.893541 12794 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:54:54.894025 12794 webserver.cc:460] Webserver started at http://127.0.98.2:38835/ using document root <none> and password file <none>
I20260927 02:54:54.894248 12794 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:54.894296 12794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:54.894413 12794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:54:54 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:54.907917 12794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "0344c7af9f234e29969c42355f486c2e"
format_stamp: "Formatted at 2026-09-27 02:54:54 on dist-test-slave-mfx9"
server_key: "da3778d930ae384841ea34f065d5bd4b"
server_key_iv: "ff146a0cef832c96e3541b940221cc59"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:54.908381 12794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0344c7af9f234e29969c42355f486c2e"
format_stamp: "Formatted at 2026-09-27 02:54:54 on dist-test-slave-mfx9"
server_key: "da3778d930ae384841ea34f065d5bd4b"
server_key_iv: "ff146a0cef832c96e3541b940221cc59"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:54.921633 12794 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.000s
I20260927 02:54:54.943627 12811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:54.944195 12794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:54:54.944296 12794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "0344c7af9f234e29969c42355f486c2e"
format_stamp: "Formatted at 2026-09-27 02:54:54 on dist-test-slave-mfx9"
server_key: "da3778d930ae384841ea34f065d5bd4b"
server_key_iv: "ff146a0cef832c96e3541b940221cc59"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:54.944396 12794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:54.976651 12794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:54.978945 12794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:54.979099 12794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:54.979327 12794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:54.979709 12794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:54.979753 12794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:54.979785 12794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:54.979830 12794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:54.988530 12794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37511
I20260927 02:54:54.989006 12794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:54:54.989745 12925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37511 every 8 connection(s)
I20260927 02:54:54.994037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 12794
I20260927 02:54:54.994139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:54:54.994308 392 external_mini_cluster.cc:1428] Setting key f01d52f31a8412626bc01eda4fff9761
Sep 27 02:54:54 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:55.000847 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:54.990817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38259 (local address 127.0.98.62:42911)
0927 02:54:54.990871 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:54.990873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:54.990903 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:54:54.991558 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:54.991559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:54.991573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:54.991651 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:54.992348 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:54.993162 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:54.994672 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:54.995001 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:54.995051 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:54.997450 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:54.997451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:54.997452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:54.997467 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:54.999189 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:54.999681 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:54.999683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:54.999683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:54.999757 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:55.000053 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:55.000054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:55.000055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:55.000201 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:55.000286 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:55.000642 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 02:54:55.000753 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:54:55.000995 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:54.990487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42911 (local address 127.0.98.2:38259)
0927 02:54:54.990678 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:54.990726 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:54.991448 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:54.991731 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:54.991735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:54.991963 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:54.992239 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:54.992244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:54.993269 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:54.993270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:54.994530 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:54.994532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:54.995218 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:54.995219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:54.995360 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:54.995906 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:54.995916 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:54.997351 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:54.999295 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:54.999302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:54.999304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:54.999587 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:54.999843 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:54.999843 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:54.999844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:54.999977 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:55.000280 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:55.000286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:55.000460 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 02:54:55.000577 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:54:55.000770 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260927 02:54:55.001432 12926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42911
I20260927 02:54:55.001552 12926 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:55.001756 12926 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:55.002195 12576 ts_manager.cc:194] Registered new tserver with Master: 0344c7af9f234e29969c42355f486c2e (127.0.98.2:37511)
I20260927 02:54:55.002676 12576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38259
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:54:55.019675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42911
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:55.108249 12931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:55.108436 12931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:55.108466 12931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:55.109576 12931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:55.109616 12931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:55.109650 12931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:54:55.111613 12931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:55.111970 12931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:55.112234 12931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:55.118233 12936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.118223 12937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.118630 12939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:55.119743 12931 server_base.cc:1034] running on GCE node
I20260927 02:54:55.119926 12931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:55.120183 12931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:55.121343 12931 hybrid_clock.cc:648] HybridClock initialized: now 1790477695121322 us; error 32 us; skew 500 ppm
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:55.124552 12931 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:54:55.125073 12931 webserver.cc:460] Webserver started at http://127.0.98.3:45045/ using document root <none> and password file <none>
I20260927 02:54:55.125417 12931 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:55.125475 12931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:55.125599 12931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:55.135886 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "fab097ea428749dbade3b6a910189de9"
format_stamp: "Formatted at 2026-09-27 02:54:55 on dist-test-slave-mfx9"
server_key: "6abcde2fb287df35fd3107207b0e6b88"
server_key_iv: "28e319a7432c083f665fba9b3189fe1e"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:55.136315 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fab097ea428749dbade3b6a910189de9"
format_stamp: "Formatted at 2026-09-27 02:54:55 on dist-test-slave-mfx9"
server_key: "6abcde2fb287df35fd3107207b0e6b88"
server_key_iv: "28e319a7432c083f665fba9b3189fe1e"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:55.137763 12931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:54:55.155712 12946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.155948 12931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:54:55.156064 12931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "fab097ea428749dbade3b6a910189de9"
format_stamp: "Formatted at 2026-09-27 02:54:55 on dist-test-slave-mfx9"
server_key: "6abcde2fb287df35fd3107207b0e6b88"
server_key_iv: "28e319a7432c083f665fba9b3189fe1e"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:55.156198 12931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:55.234169 12931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:55.236624 12931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:55.236771 12931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:55.237026 12931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:55.237402 12931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:55.237464 12931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.237501 12931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:55.237522 12931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.245343 12931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38789
I20260927 02:54:55.245805 12931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:54:55.246505 13059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38789 every 8 connection(s)
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:55.255890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 12931
I20260927 02:54:55.255985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:54:55.256188 392 external_mini_cluster.cc:1428] Setting key 4096f40598adf51fd71b2d0a512441a2
I20260927 02:54:55.257246 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.246580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36327 (local address 127.0.98.62:42911)
0927 02:54:55.246628 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:55.246630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:55.246891 (+ 261us) server_negotiation.cc:407] Connection header received
0927 02:54:55.247573 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:55.247574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:55.247588 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:55.247619 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:55.248502 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.249270 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.250841 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.251128 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.251218 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:55.253740 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:55.253742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:55.253743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:55.253751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:55.255533 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:55.256043 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:55.256044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:55.256045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:55.256109 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:55.256422 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:55.256423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:55.256423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:55.256581 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:55.256631 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:55.257058 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 02:54:55.257152 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:54:55.257797 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.246572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42911 (local address 127.0.98.3:36327)
0927 02:54:55.246732 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:55.246802 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:55.247445 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:55.247815 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:55.247819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:55.248042 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:55.248326 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:55.248332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.249380 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.249381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:55.250728 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:55.250731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.251234 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.251235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:55.251373 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:55.251939 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:55.251949 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:55.253612 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:55.255639 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:55.255646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:55.255648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:55.255939 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:55.256189 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:55.256190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:55.256190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:55.256329 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:55.256684 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:55.256691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:55.256877 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 02:54:55.257389 (+ 512us) client_negotiation.cc:240] Negotiation successful
0927 02:54:55.257569 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260927 02:54:55.258275 13060 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42911
I20260927 02:54:55.258373 13060 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:55.258584 13060 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:55.259020 12576 ts_manager.cc:194] Registered new tserver with Master: fab097ea428749dbade3b6a910189de9 (127.0.98.3:38789)
I20260927 02:54:55.259487 12576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36327
I20260927 02:54:55.267851 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:54:55.268625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 12657
I20260927 02:54:55.275071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 12794
I20260927 02:54:55.280951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 12931
I20260927 02:54:55.286581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 12544
I20260927 02:54:55.294319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42911
--webserver_interface=127.0.98.62
--webserver_port=38169
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42911
--ranger_config_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33409/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33409/kms with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:55.372985 13063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:55.373172 13063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:55.373191 13063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:55.374281 13063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:54:55.374322 13063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:55.374334 13063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:55.374346 13063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:54:55.374368 13063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:54:55.375912 13063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33409/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42911
--ranger_config_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:55.376120 13063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:55.376309 13063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:55.380784 13071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.380833 13069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.380784 13068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:55.380939 13063 server_base.cc:1034] running on GCE node
I20260927 02:54:55.381191 13063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:55.381448 13063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:55.382584 13063 hybrid_clock.cc:648] HybridClock initialized: now 1790477695382571 us; error 26 us; skew 500 ppm
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:55.384860 13063 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:54:55.385218 13063 webserver.cc:460] Webserver started at http://127.0.98.62:38169/ using document root <none> and password file <none>
I20260927 02:54:55.385380 13063 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:55.385423 13063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:55.443240 13063 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 02:54:55.443955 13079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.444136 13063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:54:55.444235 13063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "c8adf49c4ee24d57bba2c7a896474931"
format_stamp: "Formatted at 2026-09-27 02:54:53 on dist-test-slave-mfx9"
server_key: "7662328441da52b09e385660c8fe8f60"
server_key_iv: "208948a792ff66c0febbfb3724d956bc"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:55.444585 13063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:55.462700 13063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:55.464313 13063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:55.464442 13063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:55.468182 13063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42911
I20260927 02:54:55.468222 13131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42911 every 8 connection(s)
I20260927 02:54:55.468557 13063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:54:55.469342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 13063
I20260927 02:54:55.469600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46231
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42911
--webserver_port=37423
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33409/kms with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:54:55.469900 13132 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:54:55.470713 13132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Bootstrap starting.
I20260927 02:54:55.478235 13132 log.cc:824] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:55.479804 13132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:54:55.480038 13132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Bootstrap complete.
I20260927 02:54:55.482103 13132 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } }
I20260927 02:54:55.482318 13132 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8adf49c4ee24d57bba2c7a896474931, State: Initialized, Role: FOLLOWER
I20260927 02:54:55.482456 13132 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } }
I20260927 02:54:55.482533 13132 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:54:55.482580 13132 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:54:55.482623 13132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:54:55.483577 13132 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } }
I20260927 02:54:55.483666 13132 leader_election.cc:302] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8adf49c4ee24d57bba2c7a896474931; no voters:
I20260927 02:54:55.483821 13132 leader_election.cc:288] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:54:55.483868 13136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:54:55.483968 13136 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [term 2 LEADER]: Becoming Leader. State: Replica: c8adf49c4ee24d57bba2c7a896474931, State: Running, Role: LEADER
I20260927 02:54:55.484053 13132 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:54:55.484053 13136 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } }
I20260927 02:54:55.484472 13137 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8adf49c4ee24d57bba2c7a896474931. Latest consensus state: current_term: 2 leader_uuid: "c8adf49c4ee24d57bba2c7a896474931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } } }
I20260927 02:54:55.484550 13137 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:55.484866 13141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:54:55.485348 13141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:54:55.485473 13141 catalog_manager.cc:1259] Loaded cluster ID: b730be613b904469a50af613b863e054
I20260927 02:54:55.485508 13141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:54:55.485713 13132 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:54:55.484890 13138 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8adf49c4ee24d57bba2c7a896474931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8adf49c4ee24d57bba2c7a896474931" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42911 } } }
I20260927 02:54:55.487253 13138 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:55.488962 13141 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:54:55.489403 13141 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Loaded TSK: 0
I20260927 02:54:55.489666 13141 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:54:55.551484 13134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:55.551666 13134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:55.551694 13134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:55.552788 13134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:55.552826 13134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:55.552860 13134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:54:55.554659 13134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37423
--tserver_master_addrs=127.0.98.62:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:55.554898 13134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:55.555132 13134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:55.561461 13169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.561477 13168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.561792 13171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:55.562016 13134 server_base.cc:1034] running on GCE node
I20260927 02:54:55.562227 13134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:55.562445 13134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:55.563582 13134 hybrid_clock.cc:648] HybridClock initialized: now 1790477695563569 us; error 22 us; skew 500 ppm
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:55.566015 13134 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:54:55.566442 13134 webserver.cc:460] Webserver started at http://127.0.98.1:37423/ using document root <none> and password file <none>
I20260927 02:54:55.566672 13134 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:55.566735 13134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:55.591828 13134 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:54:55.598869 13180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.599030 13134 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 02:54:55.599151 13134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "4680eafcf8d744d88e6b536d947accbc"
format_stamp: "Formatted at 2026-09-27 02:54:54 on dist-test-slave-mfx9"
server_key: "1402d98562d7a3e6721c966b61c70355"
server_key_iv: "5bcd2d27477a8cf5e578facab076c368"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:55.599524 13134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:55.611102 13134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:55.613003 13134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:55.613127 13134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:55.613332 13134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:55.613643 13134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:55.613686 13134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.613714 13134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:55.613734 13134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.620468 13134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46231
I20260927 02:54:55.620949 13134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:54:55.622851 13293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46231 every 8 connection(s)
I20260927 02:54:55.629071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 13134
I20260927 02:54:55.629323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37511
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42911
--webserver_port=38835
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33409/kms with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:55.636894 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.624672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49665 (local address 127.0.98.62:42911)
0927 02:54:55.626883 (+ 2211us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:55.626890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:55.626918 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:54:55.627036 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:55.627041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:55.627080 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:55.627191 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:55.627733 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.628592 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.630190 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.630787 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.630895 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:55.633115 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:55.633120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:55.633120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:55.633131 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:55.634947 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:55.635405 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:55.635408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:55.635408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:55.635517 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:55.635807 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:55.635808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:55.635809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:55.636082 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:55.636139 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:55.636535 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 02:54:55.636648 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2164,"thread_start_us":66,"threads_started":1}
I20260927 02:54:55.637408 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.624274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42911 (local address 127.0.98.1:49665)
0927 02:54:55.624430 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:55.624470 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:55.625009 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:55.627256 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:55.627260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:55.627447 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:55.627625 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:55.627630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.628684 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.628685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:55.630087 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:55.630090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.630868 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.630869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:55.630986 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:55.631488 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:55.631498 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:55.632977 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:55.635032 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:55.635038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:55.635040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:55.635311 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:55.635592 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:55.635593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:55.635593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:55.635727 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:55.636186 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:55.636191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:55.636345 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 02:54:55.637042 (+ 697us) client_negotiation.cc:240] Negotiation successful
0927 02:54:55.637189 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260927 02:54:55.638147 13294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42911
I20260927 02:54:55.638227 13294 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:55.638403 13294 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
W20260927 02:54:55.639290 13294 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42911 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:54:55.639366 13294 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:55.639428 13294 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:55.639775 13294 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:55.639854 13294 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
W20260927 02:54:55.640164 13294 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:54:55.738507 13298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:55.738705 13298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:55.738734 13298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:55.740149 13298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:55.740212 13298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:55.740250 13298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:54:55.742311 13298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38835
--tserver_master_addrs=127.0.98.62:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:55.742534 13298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:55.742830 13298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:55.748032 13303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.748103 13304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.748287 13306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:55.748744 13298 server_base.cc:1034] running on GCE node
I20260927 02:54:55.748948 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:55.749178 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:55.750382 13298 hybrid_clock.cc:648] HybridClock initialized: now 1790477695750361 us; error 32 us; skew 500 ppm
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:55.752874 13298 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:54:55.753327 13298 webserver.cc:460] Webserver started at http://127.0.98.2:38835/ using document root <none> and password file <none>
I20260927 02:54:55.753590 13298 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:55.753695 13298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:55.781383 13298 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:54:55.784956 13314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.785099 13298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260927 02:54:55.785180 13298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "0344c7af9f234e29969c42355f486c2e"
format_stamp: "Formatted at 2026-09-27 02:54:54 on dist-test-slave-mfx9"
server_key: "da3778d930ae384841ea34f065d5bd4b"
server_key_iv: "ff146a0cef832c96e3541b940221cc59"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:55.785521 13298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:55.804188 13298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:55.806299 13298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:55.806444 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:55.806643 13298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:55.807018 13298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:55.807055 13298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.807096 13298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:55.807121 13298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.813438 13298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37511
I20260927 02:54:55.813807 13298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:54:55.814188 13427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37511 every 8 connection(s)
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:55.821327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 13298
I20260927 02:54:55.821583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38789
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42911
--webserver_port=45045
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33409/kms with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:54:55.822292 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.814475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42911 (local address 127.0.98.62:42911)
0927 02:54:55.814517 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:55.814520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:55.814822 (+ 302us) server_negotiation.cc:407] Connection header received
0927 02:54:55.815328 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:55.815330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:55.815342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:55.815382 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:55.815937 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.816431 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.817444 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.817636 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.817697 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:55.819560 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:55.819561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:55.819561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:55.819569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:55.820879 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:55.821279 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:55.821280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:55.821281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:55.821337 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:55.821606 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:55.821607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:55.821607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:55.821732 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:55.821770 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:55.822117 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 02:54:55.822208 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:54:55.822710 13430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.814483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42911 (local address 127.0.98.2:42911)
0927 02:54:55.814705 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:55.814735 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:55.815242 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:55.815450 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:55.815454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:55.815629 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:55.815860 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:55.815864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.816507 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.816508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:55.817372 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:55.817374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:55.817714 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:55.817715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:55.817851 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:55.818287 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:55.818296 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:55.819481 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:55.820945 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:55.820950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:55.820952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:55.821191 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:55.821407 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:55.821408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:55.821409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:55.821535 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:55.821812 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:55.821817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:55.821982 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 02:54:55.822367 (+ 385us) client_negotiation.cc:240] Negotiation successful
0927 02:54:55.822509 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":69,"threads_started":1}
I20260927 02:54:55.823575 13428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42911
I20260927 02:54:55.823660 13428 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:55.823827 13428 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
W20260927 02:54:55.824438 13428 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42911 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:54:55.824492 13428 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:55.824541 13428 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:55.824824 13428 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:55.824887 13428 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
W20260927 02:54:55.825148 13428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:54:55.921514 13431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:55.921849 13431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:55.921932 13431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:55.923635 13431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:55.923687 13431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:55.923723 13431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:54:55.925769 13431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45045
--tserver_master_addrs=127.0.98.62:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:55.925997 13431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:55.926280 13431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:55.932300 13439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.932976 13436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:55.933549 13437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:55.933806 13431 server_base.cc:1034] running on GCE node
I20260927 02:54:55.934094 13431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:55.934355 13431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:55.935518 13431 hybrid_clock.cc:648] HybridClock initialized: now 1790477695935486 us; error 45 us; skew 500 ppm
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:55.938740 13431 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:54:55.939286 13431 webserver.cc:460] Webserver started at http://127.0.98.3:45045/ using document root <none> and password file <none>
I20260927 02:54:55.939532 13431 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:55.939612 13431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:54:55 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:55.963307 13431 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.001s
I20260927 02:54:55.970880 13448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.971084 13431 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.001s
I20260927 02:54:55.971185 13431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "fab097ea428749dbade3b6a910189de9"
format_stamp: "Formatted at 2026-09-27 02:54:55 on dist-test-slave-mfx9"
server_key: "6abcde2fb287df35fd3107207b0e6b88"
server_key_iv: "28e319a7432c083f665fba9b3189fe1e"
server_key_version: "kuduclusterkey@0"
I20260927 02:54:55.971560 13431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:55.988284 13431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:55.990474 13431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:55.990617 13431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:55.990863 13431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:54:55.991204 13431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:54:55.991250 13431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.991277 13431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:54:55.991341 13431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:55.997834 13431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38789
I20260927 02:54:55.998203 13431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:54:55.998974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 13431
I20260927 02:54:55.999001 13561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38789 every 8 connection(s)
Sep 27 02:54:56 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:56.008762 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.998873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48757 (local address 127.0.98.62:42911)
0927 02:54:55.998919 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:55.998921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:55.999402 (+ 481us) server_negotiation.cc:407] Connection header received
0927 02:54:55.999903 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:55.999904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:55.999915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:55.999963 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:56.000476 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:56.001020 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:56.002433 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:56.002712 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:56.002776 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:56.005452 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:56.005453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:56.005453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:56.005459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:56.007208 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:56.007640 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:56.007641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:56.007642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:56.007702 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:56.007948 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:56.007949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:56.007949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:56.008084 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:56.008146 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:56.008600 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 02:54:56.008676 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:54:56.008937 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.998900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42911 (local address 127.0.98.3:48757)
0927 02:54:55.999229 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:55.999325 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:55.999810 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:56.000036 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:56.000039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:56.000212 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:56.000392 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:56.000396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:56.001122 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:56.001123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:56.002334 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:56.002336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:56.003203 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:56.003204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:56.003341 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:56.003817 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:56.003828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:56.005356 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:56.007294 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:56.007301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:56.007303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:56.007577 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:56.007770 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:56.007770 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:56.007771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:56.007899 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:56.008291 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:56.008298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:56.008467 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 02:54:56.008583 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 02:54:56.008721 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":72,"threads_started":1}
I20260927 02:54:56.009374 13562 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42911
I20260927 02:54:56.009495 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:55.999338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.0.98.62:42911)
0927 02:54:56.001415 (+ 2077us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:56.001417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:56.001567 (+ 150us) server_negotiation.cc:407] Connection header received
0927 02:54:56.001667 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:56.001668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:56.001680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:56.001723 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:56.002506 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:56.003308 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:56.004073 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:56.004275 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:56.004330 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:56.005464 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:56.005466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:56.005466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:56.005473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:56.007405 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:56.007964 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:56.007965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:56.007965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:56.008084 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:56.008281 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:56.008282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:56.008283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:56.008392 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:56.008535 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:56.009323 (+ 788us) server_negotiation.cc:299] Negotiation successful
0927 02:54:56.009404 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2057,"thread_start_us":51,"threads_started":1}
I20260927 02:54:56.009512 13562 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:56.009681 13562 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
W20260927 02:54:56.010353 13562 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42911 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:54:56.010416 13562 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:56.010484 13562 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:56.010819 13562 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:56.010885 13562 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
W20260927 02:54:56.011168 13562 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:54:56 dist-test-slave-mfx9 krb5kdc[12100](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:56.640326 13294 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:56.640467 13294 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:56.640908 13095 ts_manager.cc:194] Registered new tserver with Master: 4680eafcf8d744d88e6b536d947accbc (127.0.98.1:46231)
I20260927 02:54:56.641813 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49665
I20260927 02:54:56.825335 13428 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:56.825490 13428 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:56.825963 13095 ts_manager.cc:194] Registered new tserver with Master: 0344c7af9f234e29969c42355f486c2e (127.0.98.2:37511)
I20260927 02:54:56.826529 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42911
Sep 27 02:54:56 dist-test-slave-mfx9 krb5kdc[12100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:54:57.011336 13562 heartbeater.cc:464] Registering TS with master...
I20260927 02:54:57.011498 13562 heartbeater.cc:511] Master 127.0.98.62:42911 requested a full tablet report, sending...
I20260927 02:54:57.011830 13095 ts_manager.cc:194] Registered new tserver with Master: fab097ea428749dbade3b6a910189de9 (127.0.98.3:38789)
I20260927 02:54:57.012348 13095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48757
I20260927 02:54:57.012372 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:54:57.019246 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.013307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.0.98.62:42911)
0927 02:54:57.013433 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:57.013435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:57.013628 (+ 193us) server_negotiation.cc:407] Connection header received
0927 02:54:57.013696 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:57.013697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:57.013709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:57.013765 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:57.014310 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.014855 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.015535 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.015751 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.015805 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.016887 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:57.016888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:57.016889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:57.016897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:57.018220 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:57.018521 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:57.018522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:57.018522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:57.018576 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:57.018742 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:57.018742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:57.018743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:57.018874 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:57.018961 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:57.019057 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:54:57.019140 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260927 02:54:57.023733 13095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:54:57.026180 13095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:54:57.036322 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.030039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.0.98.1:46231)
0927 02:54:57.030243 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:57.030249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:57.030943 (+ 694us) server_negotiation.cc:407] Connection header received
0927 02:54:57.031004 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:57.031006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:57.031024 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:57.031118 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:57.031446 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.032652 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.034475 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.035355 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.035418 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.035500 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:57.036170 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 02:54:57.036222 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
I20260927 02:54:57.036501 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.029520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37511 (local address 127.0.0.1:51984)
0927 02:54:57.030304 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:57.030343 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:57.030410 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:57.031220 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:57.031224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:57.031227 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:57.031338 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.031340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.032815 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.032816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.034335 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.034339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.036200 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.036201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.036267 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.036286 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:54:57.036317 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 02:54:57.036399 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":27,"thread_start_us":92,"threads_started":1}
I20260927 02:54:57.036825 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.029606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46231 (local address 127.0.0.1:41824)
0927 02:54:57.030858 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:57.030868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:57.030928 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:57.031226 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:57.031228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:57.031230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:57.031338 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.031343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.033107 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.033108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.034351 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.034355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.035912 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.035913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.036024 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.036041 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:54:57.036688 (+ 647us) client_negotiation.cc:240] Negotiation successful
0927 02:54:57.036743 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"mutex_wait_us":10,"thread_start_us":87,"threads_started":1}
I20260927 02:54:57.037400 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.029794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.0.98.2:37511)
0927 02:54:57.031031 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:57.031037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:57.031046 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:54:57.031081 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:57.031083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:57.031098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:57.031397 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:57.031473 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.032711 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.034479 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.035356 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.037121 (+ 1765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.037223 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:57.037251 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 02:54:57.037299 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":45,"threads_started":1}
I20260927 02:54:57.037765 13224 tablet_service.cc:1467] Processing CreateTablet for tablet 2109e8d4560a418193795aef52983272 (DEFAULT_TABLE table=test-table [id=c93c4cb0d623402f997993f723adcd4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:54:57.038177 13224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2109e8d4560a418193795aef52983272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:57.040241 13362 tablet_service.cc:1467] Processing CreateTablet for tablet 2109e8d4560a418193795aef52983272 (DEFAULT_TABLE table=test-table [id=c93c4cb0d623402f997993f723adcd4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:54:57.040607 13362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2109e8d4560a418193795aef52983272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:57.046914 13595 tablet_bootstrap.cc:492] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc: Bootstrap starting.
I20260927 02:54:57.046905 13596 tablet_bootstrap.cc:492] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: Bootstrap starting.
I20260927 02:54:57.047385 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.029905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.0.98.3:38789)
0927 02:54:57.030614 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:57.030618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:57.035584 (+ 4966us) server_negotiation.cc:407] Connection header received
0927 02:54:57.035725 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:57.035728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:57.035745 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:57.035799 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:57.037668 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.039004 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.045559 (+ 6555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.046748 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.046857 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.046944 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:57.047242 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 02:54:57.047293 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":87,"threads_started":1}
I20260927 02:54:57.047775 13595 tablet_bootstrap.cc:654] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:57.047781 13596 tablet_bootstrap.cc:654] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:57.048012 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.029828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38789 (local address 127.0.0.1:53854)
0927 02:54:57.035511 (+ 5683us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:57.035523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:57.035629 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:57.035913 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:57.035914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:57.035916 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:57.036004 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.036007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.039110 (+ 3103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.039111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.045344 (+ 6233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.045351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.047051 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.047052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.047135 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.047140 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:54:57.047855 (+ 715us) client_negotiation.cc:240] Negotiation successful
0927 02:54:57.047927 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5661,"thread_start_us":65,"threads_started":1}
I20260927 02:54:57.048154 13596 log.cc:824] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:57.049070 13490 tablet_service.cc:1467] Processing CreateTablet for tablet 2109e8d4560a418193795aef52983272 (DEFAULT_TABLE table=test-table [id=c93c4cb0d623402f997993f723adcd4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:54:57.049486 13490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2109e8d4560a418193795aef52983272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:57.053381 13598 tablet_bootstrap.cc:492] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: Bootstrap starting.
I20260927 02:54:57.054157 13598 tablet_bootstrap.cc:654] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:57.054486 13598 log.cc:824] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:57.055528 13598 tablet_bootstrap.cc:492] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: No bootstrap required, opened a new log
I20260927 02:54:57.055542 13596 tablet_bootstrap.cc:492] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: No bootstrap required, opened a new log
I20260927 02:54:57.055627 13596 ts_tablet_manager.cc:1397] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 02:54:57.055682 13598 ts_tablet_manager.cc:1397] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:54:57.058372 13598 raft_consensus.cc:348] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.058373 13596 raft_consensus.cc:348] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.058574 13598 raft_consensus.cc:374] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:57.058631 13598 raft_consensus.cc:729] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab097ea428749dbade3b6a910189de9, State: Initialized, Role: FOLLOWER
I20260927 02:54:57.058830 13598 consensus_queue.cc:260] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.059139 13598 ts_tablet_manager.cc:1428] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:54:57.059294 13562 heartbeater.cc:503] Master 127.0.98.62:42911 was elected leader, sending a full tablet report...
I20260927 02:54:57.058578 13596 raft_consensus.cc:374] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:57.059556 13596 raft_consensus.cc:729] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0344c7af9f234e29969c42355f486c2e, State: Initialized, Role: FOLLOWER
I20260927 02:54:57.059620 13595 log.cc:824] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:57.059725 13596 consensus_queue.cc:260] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.060026 13596 ts_tablet_manager.cc:1428] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 02:54:57.060180 13428 heartbeater.cc:503] Master 127.0.98.62:42911 was elected leader, sending a full tablet report...
I20260927 02:54:57.060554 13595 tablet_bootstrap.cc:492] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc: No bootstrap required, opened a new log
I20260927 02:54:57.060622 13595 ts_tablet_manager.cc:1397] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260927 02:54:57.062655 13595 raft_consensus.cc:348] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.062870 13595 raft_consensus.cc:374] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:57.062938 13595 raft_consensus.cc:729] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4680eafcf8d744d88e6b536d947accbc, State: Initialized, Role: FOLLOWER
I20260927 02:54:57.063078 13595 consensus_queue.cc:260] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.063371 13294 heartbeater.cc:503] Master 127.0.98.62:42911 was elected leader, sending a full tablet report...
I20260927 02:54:57.063359 13595 ts_tablet_manager.cc:1428] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260927 02:54:57.065059 13429 tablet.cc:2369] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:54:57.068837 13603 raft_consensus.cc:482] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:54:57.068959 13603 raft_consensus.cc:504] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.069317 13603 leader_election.cc:288] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0344c7af9f234e29969c42355f486c2e (127.0.98.2:37511), fab097ea428749dbade3b6a910189de9 (127.0.98.3:38789)
I20260927 02:54:57.074041 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.069664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54357 (local address 127.0.98.2:37511)
0927 02:54:57.069710 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:57.069712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:57.069723 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:54:57.069827 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:57.069828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:57.069841 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:57.069876 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:57.070208 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.071271 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.072713 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.073289 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.073338 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.073350 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:57.073913 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 02:54:57.073961 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:54:57.074352 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.069479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37511 (local address 127.0.98.1:54357)
0927 02:54:57.069667 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:57.069679 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:57.069738 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:57.069979 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:57.069980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:57.069982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:57.070080 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.070083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.071380 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.071381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.072610 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.072612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.073414 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.073414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.073821 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.073827 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:54:57.074203 (+ 376us) client_negotiation.cc:240] Negotiation successful
0927 02:54:57.074249 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":111,"threads_started":1}
I20260927 02:54:57.074980 13382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2109e8d4560a418193795aef52983272" candidate_uuid: "4680eafcf8d744d88e6b536d947accbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0344c7af9f234e29969c42355f486c2e" is_pre_election: true
I20260927 02:54:57.075161 13382 raft_consensus.cc:2454] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4680eafcf8d744d88e6b536d947accbc in term 0.
I20260927 02:54:57.075168 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.070070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43785 (local address 127.0.98.3:38789)
0927 02:54:57.071225 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:57.071227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:57.071380 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:54:57.071482 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:57.071483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:57.071496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:57.071529 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:54:57.071729 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.072449 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.073835 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.074408 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.074461 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.074471 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:57.075052 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 02:54:57.075105 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130}
I20260927 02:54:57.075474 13182 leader_election.cc:302] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0344c7af9f234e29969c42355f486c2e, 4680eafcf8d744d88e6b536d947accbc; no voters:
I20260927 02:54:57.075569 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.069840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38789 (local address 127.0.98.1:43785)
0927 02:54:57.071321 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:57.071332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:57.071406 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:57.071588 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:57.071589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:57.071591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:54:57.071650 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.071652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.072555 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.072556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.073736 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:57.073740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.074876 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.074877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:57.074946 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.074960 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:54:57.075267 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 02:54:57.075456 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":79,"threads_started":1}
I20260927 02:54:57.075943 13516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2109e8d4560a418193795aef52983272" candidate_uuid: "4680eafcf8d744d88e6b536d947accbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fab097ea428749dbade3b6a910189de9" is_pre_election: true
I20260927 02:54:57.076094 13516 raft_consensus.cc:2454] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4680eafcf8d744d88e6b536d947accbc in term 0.
I20260927 02:54:57.076236 13603 raft_consensus.cc:2781] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:54:57.076344 13603 raft_consensus.cc:482] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:54:57.076387 13603 raft_consensus.cc:3037] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:57.077112 13603 raft_consensus.cc:504] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.077356 13603 leader_election.cc:288] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [CANDIDATE]: Term 1 election: Requested vote from peers 0344c7af9f234e29969c42355f486c2e (127.0.98.2:37511), fab097ea428749dbade3b6a910189de9 (127.0.98.3:38789)
I20260927 02:54:57.077358 13382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2109e8d4560a418193795aef52983272" candidate_uuid: "4680eafcf8d744d88e6b536d947accbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0344c7af9f234e29969c42355f486c2e"
I20260927 02:54:57.077435 13382 raft_consensus.cc:3037] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:57.078563 13382 raft_consensus.cc:2454] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4680eafcf8d744d88e6b536d947accbc in term 1.
I20260927 02:54:57.078748 13182 leader_election.cc:302] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0344c7af9f234e29969c42355f486c2e, 4680eafcf8d744d88e6b536d947accbc; no voters:
I20260927 02:54:57.078913 13516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2109e8d4560a418193795aef52983272" candidate_uuid: "4680eafcf8d744d88e6b536d947accbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fab097ea428749dbade3b6a910189de9"
I20260927 02:54:57.078994 13516 raft_consensus.cc:3037] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:57.079653 13516 raft_consensus.cc:2454] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4680eafcf8d744d88e6b536d947accbc in term 1.
I20260927 02:54:57.079828 13603 raft_consensus.cc:2781] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:54:57.079998 13603 raft_consensus.cc:686] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [term 1 LEADER]: Becoming Leader. State: Replica: 4680eafcf8d744d88e6b536d947accbc, State: Running, Role: LEADER
I20260927 02:54:57.080129 13603 consensus_queue.cc:237] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } }
I20260927 02:54:57.081012 13096 catalog_manager.cc:5495] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4680eafcf8d744d88e6b536d947accbc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4680eafcf8d744d88e6b536d947accbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4680eafcf8d744d88e6b536d947accbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 } health_report { overall_health: UNKNOWN } } }
I20260927 02:54:57.087438 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.085145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.0.98.1:46231)
0927 02:54:57.085198 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:57.085201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:57.085210 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:54:57.085293 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:57.085293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:57.085309 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:57.085371 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:54:57.085535 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.086000 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.086705 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:57.086959 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:57.087012 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:57.087160 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:54:57.087260 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:54:57.087329 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:57.087349 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 02:54:57.087369 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:54:57.089148 13382 raft_consensus.cc:1261] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 1 FOLLOWER]: Refusing update from remote peer 4680eafcf8d744d88e6b536d947accbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:54:57.089148 13516 raft_consensus.cc:1261] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 1 FOLLOWER]: Refusing update from remote peer 4680eafcf8d744d88e6b536d947accbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:54:57.089442 13603 consensus_queue.cc:1035] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fab097ea428749dbade3b6a910189de9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:54:57.089615 13606 consensus_queue.cc:1035] T 2109e8d4560a418193795aef52983272 P 4680eafcf8d744d88e6b536d947accbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0344c7af9f234e29969c42355f486c2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:54:57.092576 13609 mvcc.cc:205] Tried to move back new op lower bound from 7333796647274401792 to 7333796647240937472. Current Snapshot: MvccSnapshot[applied={T|T < 7333796647274401792}]
I20260927 02:54:57.093022 13610 mvcc.cc:205] Tried to move back new op lower bound from 7333796647274401792 to 7333796647240937472. Current Snapshot: MvccSnapshot[applied={T|T < 7333796647274401792}]
I20260927 02:54:57.093711 13611 mvcc.cc:205] Tried to move back new op lower bound from 7333796647274401792 to 7333796647240937472. Current Snapshot: MvccSnapshot[applied={T|T < 7333796647274401792}]
I20260927 02:54:57.095873 13096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:54:57.095963 13096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:54:57.096761 13096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8adf49c4ee24d57bba2c7a896474931: Sending DeleteTablet for 3 replicas of tablet 2109e8d4560a418193795aef52983272
I20260927 02:54:57.097244 13362 tablet_service.cc:1514] Processing DeleteTablet for tablet 2109e8d4560a418193795aef52983272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:54:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51984
I20260927 02:54:57.097996 13490 tablet_service.cc:1514] Processing DeleteTablet for tablet 2109e8d4560a418193795aef52983272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:54:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53854
I20260927 02:54:57.097888 13224 tablet_service.cc:1514] Processing DeleteTablet for tablet 2109e8d4560a418193795aef52983272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:54:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41824
I20260927 02:54:57.099066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 13134
I20260927 02:54:57.100881 13618 tablet_replica.cc:330] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: stopping tablet replica
I20260927 02:54:57.100988 13618 raft_consensus.cc:2229] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:54:57.101063 13618 raft_consensus.cc:2258] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:54:57.101265 13620 tablet_replica.cc:330] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: stopping tablet replica
I20260927 02:54:57.101403 13620 raft_consensus.cc:2229] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:54:57.101516 13620 raft_consensus.cc:2258] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:54:57.104614 13080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46231 (error 108)
I20260927 02:54:57.105104 13620 ts_tablet_manager.cc:1905] T 2109e8d4560a418193795aef52983272 P 0344c7af9f234e29969c42355f486c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:54:57.105378 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.105268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46231 (local address 127.0.0.1:41834)
0927 02:54:57.105314 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:57.105346 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:54:57.105559 13080 catalog_manager.cc:4584] TS 4680eafcf8d744d88e6b536d947accbc (127.0.98.1:46231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2109e8d4560a418193795aef52983272: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46231: connect: Connection refused (error 111)
I20260927 02:54:57.105690 13618 ts_tablet_manager.cc:1905] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:54:57.105785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 13298
I20260927 02:54:57.107347 13618 ts_tablet_manager.cc:1918] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:54:57.107406 13618 log.cc:1196] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/2109e8d4560a418193795aef52983272
I20260927 02:54:57.107626 13618 ts_tablet_manager.cc:1939] T 2109e8d4560a418193795aef52983272 P fab097ea428749dbade3b6a910189de9: Deleting consensus metadata
I20260927 02:54:57.108150 13083 catalog_manager.cc:4841] TS fab097ea428749dbade3b6a910189de9 (127.0.98.3:38789): tablet 2109e8d4560a418193795aef52983272 (table test-table [id=c93c4cb0d623402f997993f723adcd4f]) successfully deleted
I20260927 02:54:57.111260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 13431
I20260927 02:54:57.111748 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:57.111360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37511 (local address 127.0.0.1:51994)
0927 02:54:57.111691 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:57.111717 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
I20260927 02:54:57.116729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 13063
I20260927 02:54:57.121608 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:54:57.455631 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:54:57.455715 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:54:57.789465 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:54:57.789 UTC [12130] LOG: received smart shutdown request
2026-09-27 02:54:57.793 UTC [12130] LOG: background worker "logical replication launcher" (PID 12139) exited with exit code 1
2026-09-27 02:54:57.793 UTC [12134] LOG: shutting down
2026-09-27 02:54:57.942 UTC [12130] LOG: database system is shut down
2026-09-27T02:54:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45511 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:54:58 dist-test-slave-mfx9 krb5kdc[13630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:54:58 dist-test-slave-mfx9 krb5kdc[13630](info): set up 2 sockets
Sep 27 02:54:58 dist-test-slave-mfx9 krb5kdc[13630](info): commencing operation
krb5kdc: starting...
W20260927 02:54:58.990276 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:54:59.097357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34049,127.0.98.61:44655,127.0.98.60:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:54:59.176263 13643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:59.176443 13643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:59.176463 13643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:59.177560 13643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:54:59.177596 13643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:59.177609 13643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:59.177623 13643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:54:59.177645 13643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:54:59.179107 13643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34049,127.0.98.61:44655,127.0.98.60:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:59.179311 13643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:59.179503 13643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:59.183786 13649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:59.183787 13651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:59.183952 13648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:59.184026 13643 server_base.cc:1034] running on GCE node
I20260927 02:54:59.184259 13643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:59.184484 13643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:59.185613 13643 hybrid_clock.cc:648] HybridClock initialized: now 1790477699185602 us; error 27 us; skew 500 ppm
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:59.191316 13643 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:54:59.191723 13643 webserver.cc:460] Webserver started at http://127.0.98.62:44669/ using document root <none> and password file <none>
I20260927 02:54:59.191900 13643 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:59.191944 13643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:59.192060 13643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:54:59.193022 13643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "d02fc75aaa254406adfafbc628a09350"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "9bbe5046a85d21b72862ffca2368a218"
server_key_iv: "b37c48a89f055ec63036e3ee7513d0d9"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.193334 13643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "d02fc75aaa254406adfafbc628a09350"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "9bbe5046a85d21b72862ffca2368a218"
server_key_iv: "b37c48a89f055ec63036e3ee7513d0d9"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.194444 13643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:54:59.195097 13658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:59.195231 13643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:59.195308 13643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "d02fc75aaa254406adfafbc628a09350"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "9bbe5046a85d21b72862ffca2368a218"
server_key_iv: "b37c48a89f055ec63036e3ee7513d0d9"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.195363 13643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:59.211478 13643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:59.213016 13643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:59.213133 13643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:59.216601 13643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34049
I20260927 02:54:59.216635 13710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34049 every 8 connection(s)
I20260927 02:54:59.216917 13643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:54:59.217662 13711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:59.219063 13711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:54:59.220981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 13643
I20260927 02:54:59.221063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:54:59.221227 392 external_mini_cluster.cc:1428] Setting key b1947a6c82770b9d0248d5e009428832
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 02:54:59.227454 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.219643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.0.1:52800)
0927 02:54:59.219775 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.219805 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.220244 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.220477 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.220480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.220745 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.220901 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.220906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.221915 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.221916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.222556 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.222558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.223032 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.223034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.223097 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.223502 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.223510 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.224727 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.226187 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.226193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.226195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.226365 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.226549 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.226549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.226550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.226623 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.226961 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.226966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.227022 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.227100 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.227307 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":88,"thread_start_us":62,"threads_started":1}
I20260927 02:54:59.227459 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.219782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.0.98.62:34049)
0927 02:54:59.219884 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.219888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.219916 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:54:59.220382 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.220383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.220398 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.220467 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.220989 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.221837 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.222633 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.222966 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.223053 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.224798 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.224804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.224804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.224810 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.226114 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.226427 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.226430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.226431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.226497 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.226673 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.226674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.226674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.226906 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.226975 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.227129 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.227241 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":36,"threads_started":1}
I20260927 02:54:59.228137 13711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:54:59.228444 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.228340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55258)
0927 02:54:59.228380 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.228419 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:54:59.228611 13660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
I20260927 02:54:59.228940 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.228887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55268)
0927 02:54:59.228907 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.228926 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 02:54:59.229032 13711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44655: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:54:59.241736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44655
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34049,127.0.98.61:44655,127.0.98.60:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:54:59.278221 13711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } attempt: 1
I20260927 02:54:59.278719 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.278590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55274)
0927 02:54:59.278643 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.278691 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:54:59.279069 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.278991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55282)
0927 02:54:59.279029 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.279050 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:54:59.279165 13711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44655: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
W20260927 02:54:59.318487 13718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:59.318673 13718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:59.318702 13718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:59.319835 13718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:54:59.319876 13718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:59.319895 13718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:59.319911 13718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:54:59.319924 13718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:54:59.321398 13718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34049,127.0.98.61:44655,127.0.98.60:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:59.321633 13718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:59.321825 13718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:54:59.326272 13718 server_base.cc:1034] running on GCE node
W20260927 02:54:59.326249 13726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:59.326234 13723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:59.326249 13724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:59.326684 13718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:59.327011 13718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:59.328153 13718 hybrid_clock.cc:648] HybridClock initialized: now 1790477699328131 us; error 34 us; skew 500 ppm
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:59.330344 13718 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:54:59.330713 13718 webserver.cc:460] Webserver started at http://127.0.98.61:36577/ using document root <none> and password file <none>
I20260927 02:54:59.330935 13718 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:59.330986 13718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:59.331099 13718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:54:59.332065 13718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/data/instance:
uuid: "9f8c383c8306426390cc7bb55bd583a0"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "93883affcce3835227b41e7a08eb621e"
server_key_iv: "cbe1d691f2bcaf9edc9747f82663093f"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.332358 13718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/wal/instance:
uuid: "9f8c383c8306426390cc7bb55bd583a0"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "93883affcce3835227b41e7a08eb621e"
server_key_iv: "cbe1d691f2bcaf9edc9747f82663093f"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.333477 13718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:54:59.334080 13733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:59.334228 13718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:54:59.334303 13718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/wal
uuid: "9f8c383c8306426390cc7bb55bd583a0"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "93883affcce3835227b41e7a08eb621e"
server_key_iv: "cbe1d691f2bcaf9edc9747f82663093f"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.334359 13718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:59.347291 13711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } attempt: 2
I20260927 02:54:59.347373 13718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:59.347641 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.347507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55288)
0927 02:54:59.347571 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.347607 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:54:59.347934 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.347849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55304)
0927 02:54:59.347887 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.347913 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:54:59.348027 13711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44655: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44655: connect: Connection refused (error 111)
I20260927 02:54:59.349112 13718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:59.349208 13718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:59.352769 13718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44655
I20260927 02:54:59.352794 13785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44655 every 8 connection(s)
I20260927 02:54:59.353091 13718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:54:59.353803 13786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:59.355221 13786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:54:59.355437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 13718
I20260927 02:54:59.355517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-1/wal/instance
I20260927 02:54:59.355702 392 external_mini_cluster.cc:1428] Setting key b9a210d5e6c9a9780d9e345022c14834
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 02:54:59.364157 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.355929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.0.98.62:34049)
0927 02:54:59.355981 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.355983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.356169 (+ 186us) server_negotiation.cc:407] Connection header received
0927 02:54:59.356596 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.356597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.356608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.356675 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.357400 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.357859 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.358888 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.359196 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.359264 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.361402 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.361404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.361404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.361411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.362860 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.363263 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.363265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.363265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.363316 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.363496 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.363496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.363497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.363589 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.363634 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.364023 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.364095 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:54:59.364252 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.355899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.0.1:52810)
0927 02:54:59.356051 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.356090 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.356521 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.356734 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.356738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.357058 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.357320 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.357325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.357948 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.357949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.358785 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.358788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.359268 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.359269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.359369 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.359834 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.359842 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.361312 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.362936 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.362943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.362945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.363193 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.363366 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.363367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.363367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.363453 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.363668 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.363673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.363854 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.363963 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.364083 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260927 02:54:59.364809 13786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:54:59.371932 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.365011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55308)
0927 02:54:59.365238 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.365249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.365294 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.365425 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.365426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.365556 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.365596 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.365598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.366426 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.366427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.367170 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.367173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.367713 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.367714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.367776 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.368105 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.368110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.369274 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.371079 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.371080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.371080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.371222 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.371419 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.371419 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.371420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.371460 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.371742 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.371743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.371773 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.371814 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.371874 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 02:54:59.372033 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.365093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.0.98.61:44655)
0927 02:54:59.365246 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.365250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.365293 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:54:59.365343 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.365345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.365359 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.365420 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.365657 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.366344 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.367246 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.367644 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.367715 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.369354 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.369363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.369363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.369370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.370957 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.371289 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.371291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.371292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.371362 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.371509 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.371509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.371510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.371681 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.371758 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.371850 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.371939 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":125,"thread_start_us":101,"threads_started":1}
I20260927 02:54:59.372606 13786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:54:59.372885 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.372795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46014)
0927 02:54:59.372831 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.372869 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:54:59.373013 13737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
I20260927 02:54:59.373394 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.373288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46022)
0927 02:54:59.373358 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.373380 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 02:54:59.373487 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39773: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:54:59.379488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39773
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34049,127.0.98.61:44655,127.0.98.60:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:54:59.422696 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } attempt: 1
I20260927 02:54:59.423269 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.423127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46028)
0927 02:54:59.423189 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.423238 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:54:59.423612 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.423523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46030)
0927 02:54:59.423565 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.423592 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:54:59.423717 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39773: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
I20260927 02:54:59.439162 13711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } attempt: 3
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:54:59.446866 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.439601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.0.98.61:44655)
0927 02:54:59.439652 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.439654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.439667 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:54:59.439774 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.439775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.439788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.439836 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.440225 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.440746 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.441774 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.442043 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.442109 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.444407 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.444409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.444409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.444417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.445708 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.446076 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.446077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.446077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.446129 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.446336 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.446337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.446337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.446470 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.446529 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.446700 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.446797 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:54:59.446893 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.439542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55318)
0927 02:54:59.439598 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.439614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.439687 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.439862 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.439863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.440069 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.440152 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.440155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.440842 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.440843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.441675 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.441679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.442190 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.442191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.442322 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.442911 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.442917 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.444305 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.445791 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.445792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.445793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.446001 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.446187 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.446188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.446188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.446254 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.446551 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.446552 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.446604 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.446675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.446798 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:54:59.447392 13711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:54:59.447698 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.447605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46046)
0927 02:54:59.447639 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.447673 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 02:54:59.447989 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.447908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46056)
0927 02:54:59.447947 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.447972 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:54:59.448083 13711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39773: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
W20260927 02:54:59.461009 13793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:54:59.461199 13793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:54:59.461220 13793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:54:59.462317 13793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:54:59.462355 13793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:54:59.462368 13793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:54:59.462378 13793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:54:59.462389 13793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:54:59.463891 13793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34049,127.0.98.61:44655,127.0.98.60:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:54:59.464107 13793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:54:59.464323 13793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:54:59.468677 13801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:59.468699 13799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:54:59.468690 13798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:54:59.468719 13793 server_base.cc:1034] running on GCE node
I20260927 02:54:59.469055 13793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:54:59.469308 13793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:54:59.470433 13793 hybrid_clock.cc:648] HybridClock initialized: now 1790477699470421 us; error 29 us; skew 500 ppm
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:54:59.472764 13793 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:54:59.473167 13793 webserver.cc:460] Webserver started at http://127.0.98.60:45297/ using document root <none> and password file <none>
I20260927 02:54:59.473361 13793 fs_manager.cc:359] Metadata directory not provided
I20260927 02:54:59.473402 13793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:54:59.473518 13793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:54:59.474483 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/data/instance:
uuid: "b5d20bb702d947c598db2b677f893a78"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "bca5a4750ef1522ef62aa7a431203603"
server_key_iv: "1cb94b2106bda2c017077c62fdb00d85"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.474843 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/wal/instance:
uuid: "b5d20bb702d947c598db2b677f893a78"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "bca5a4750ef1522ef62aa7a431203603"
server_key_iv: "1cb94b2106bda2c017077c62fdb00d85"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.475968 13793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:54:59.476567 13808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:54:59.476706 13793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:54:59.476775 13793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/wal
uuid: "b5d20bb702d947c598db2b677f893a78"
format_stamp: "Formatted at 2026-09-27 02:54:59 on dist-test-slave-mfx9"
server_key: "bca5a4750ef1522ef62aa7a431203603"
server_key_iv: "1cb94b2106bda2c017077c62fdb00d85"
server_key_version: "encryptionkey@0"
I20260927 02:54:59.476838 13793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:54:59.479211 13711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } attempt: 1
I20260927 02:54:59.479550 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.479418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46058)
0927 02:54:59.479475 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.479513 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:54:59.479842 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.479759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46062)
0927 02:54:59.479794 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.479821 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:54:59.479925 13711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39773: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
I20260927 02:54:59.491269 13793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:54:59.491878 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } attempt: 2
I20260927 02:54:59.492230 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.492105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46064)
0927 02:54:59.492168 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.492197 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:54:59.492609 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.492527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46072)
0927 02:54:59.492560 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.492585 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 02:54:59.492704 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39773: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39773: connect: Connection refused (error 111)
I20260927 02:54:59.493426 13793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:54:59.493547 13793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:54:59.497056 13793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39773
I20260927 02:54:59.497095 13860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39773 every 8 connection(s)
I20260927 02:54:59.497390 13793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:54:59.498157 13861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:54:59.499715 13861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:54:59.503368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 13793
I20260927 02:54:59.503463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477553417353-392-0/minicluster-data/master-2/wal/instance
I20260927 02:54:59.503669 392 external_mini_cluster.cc:1428] Setting key 968f8e5f24db7804dc008d8e1b0a1c29
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:54:59.508395 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.500323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.0.98.62:34049)
0927 02:54:59.500386 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.500388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.500555 (+ 167us) server_negotiation.cc:407] Connection header received
0927 02:54:59.501051 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.501053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.501064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.501114 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.501681 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.502148 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.503136 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.503320 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.503376 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.505529 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.505530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.505531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.505538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.506985 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.507404 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.507405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.507405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.507471 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.507739 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.507741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.507741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.507880 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.507952 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.508222 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.508299 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:54:59.508440 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.500290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.0.1:52824)
0927 02:54:59.500443 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.500475 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.500944 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.501177 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.501180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.501405 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.501594 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.501598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.502221 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.502222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.503059 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.503061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.503383 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.503384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.503531 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.503983 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.503991 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.505460 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.507061 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.507068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.507070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.507312 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.507531 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.507532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.507532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.507666 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.507955 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.507959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.508082 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.508177 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.508288 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260927 02:54:59.509099 13861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:54:59.512418 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.504224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.0.98.60:39773)
0927 02:54:59.504671 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.504675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.504690 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:54:59.504705 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.504707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.504721 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.504779 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.505414 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.506253 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.506985 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.507282 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.507347 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.509820 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.509826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.509826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.509832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.511399 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.511684 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.511687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.511687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.511740 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.511898 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.511899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.511899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.512022 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.512154 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.512235 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.512316 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":63,"threads_started":1}
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:54:59.515585 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.509307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44655 (local address 127.0.0.1:55328)
0927 02:54:59.509405 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.509415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.509453 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.509616 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.509617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.509731 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.509778 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.509779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.510538 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.510538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.511332 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.511335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.511659 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.511660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.511726 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.512053 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.512057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.513259 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.514721 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.514721 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.514722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.514893 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.515070 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.515071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.515071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.515116 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.515338 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.515338 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.515374 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.515426 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.515503 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 02:54:59.515618 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.509367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.0.98.61:44655)
0927 02:54:59.509407 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.509409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.509497 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:54:59.509520 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.509521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.509529 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.509600 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.509874 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.510449 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.511408 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.511593 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.511653 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.513331 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.513332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.513333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.513339 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.514646 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.514946 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.514947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.514947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.515001 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.515176 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.515178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.515178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.515286 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.515339 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.515453 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.515516 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:54:59.515900 13861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:54:59.520040 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.513125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.0.98.62:34049)
0927 02:54:59.513156 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.513158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.513164 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:54:59.513181 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.513182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.513189 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.513249 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.513708 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.514273 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.515120 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.515316 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.515365 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.517348 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.517349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.517349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.517356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.518615 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.518969 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.518970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.518970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.519012 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.519184 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.519185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.519185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.519282 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.519846 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.519885 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.519951 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:54:59.521751 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.516126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46088)
0927 02:54:59.516153 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.516161 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.516195 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.516289 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.516290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.516402 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.516424 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.516425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.516988 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.516988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.517523 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.517525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.517897 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.517898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.517942 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.518341 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.518345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.519449 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.520959 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.520960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.520961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.521140 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.521307 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.521307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.521308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.521351 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.521552 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.521553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.521582 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.521623 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.521697 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 02:54:59.521847 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.516158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.0.98.60:39773)
0927 02:54:59.516202 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.516203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.516212 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:54:59.516231 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.516232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.516240 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.516279 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.516478 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.516926 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.517600 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.517820 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.517875 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.519520 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.519521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.519521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.519527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.520895 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.521196 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.521197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.521197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.521251 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.521396 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.521396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.521396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.521503 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.521552 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.521654 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.521749 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":30}
I20260927 02:54:59.523464 13861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78: Bootstrap starting.
I20260927 02:54:59.524240 13861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78: Neither blocks nor log segments found. Creating new log.
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:54:59.524493 13861 log.cc:824] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:59.525219 13861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78: No bootstrap required, opened a new log
I20260927 02:54:59.527016 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.520568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.0.98.61:44655)
0927 02:54:59.520600 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.520602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.520610 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:54:59.520629 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.520629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.520637 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.520708 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.521116 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.521740 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.522434 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.522690 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.522752 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.524651 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.524653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.524653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.524660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.526021 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.526304 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.526305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.526305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.526361 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.526496 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.526496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.526497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.526621 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.526820 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.526858 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.526929 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:54:59.527179 13861 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.527323 13861 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:59.527365 13861 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d20bb702d947c598db2b677f893a78, State: Initialized, Role: FOLLOWER
I20260927 02:54:59.527482 13861 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.527720 13867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.527783 13861 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:54:59.527809 13867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:54:59.528319 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 02:54:59.529381 13878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:54:59.529480 13878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:54:59.529680 13793 master_runner.cc:186] Error getting master registration for 127.0.98.62:34049: OK, instance_id { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" instance_seqno: 1790477699215637 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:54:59.531872 13793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:54:59.537788 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.529629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.98.62:34049)
0927 02:54:59.529665 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.529667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.529713 (+ 46us) server_negotiation.cc:407] Connection header received
0927 02:54:59.529781 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.529782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.529790 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.529843 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.530529 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.531268 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.532272 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.532568 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.532611 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.534039 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.534041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.534041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.534048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.535821 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.536893 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.536894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.536894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.536951 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.537197 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.537198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.537198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.537325 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.537395 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.537610 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.537691 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:54:59.538187 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.529912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.0.98.60:39773)
0927 02:54:59.529944 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.529945 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.530463 (+ 518us) server_negotiation.cc:407] Connection header received
0927 02:54:59.530546 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.530547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.530555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.530580 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.531266 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.531909 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.532864 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.533113 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.533151 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.534370 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.534371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.534371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.534378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.536497 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.537290 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.537291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.537292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.537351 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.537545 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.537546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.537546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.537703 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.537784 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.538021 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.538092 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:54:59.539125 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.530223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.0.98.61:44655)
0927 02:54:59.530253 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.530255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.532108 (+ 1853us) server_negotiation.cc:407] Connection header received
0927 02:54:59.532208 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.532209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.532219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.532246 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.532866 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.534023 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.535117 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.535383 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.535441 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.536381 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.536382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.536382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.536389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.537820 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.538160 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.538161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.538161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.538212 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.538540 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.538541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.538541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.538669 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.538744 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.538975 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.539042 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:54:59.555076 13711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } attempt: 2
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:54:59.561574 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.555425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.98.60:39773)
0927 02:54:59.555480 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.555482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.555510 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:54:59.555530 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.555531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.555548 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.555596 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.555834 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.556573 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.557170 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.557469 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.557527 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.559188 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.559190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.559190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.559197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.560643 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.560997 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.560998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.560998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.561040 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.561186 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.561187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.561187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.561270 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.561311 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.561460 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.561513 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:54:59.561625 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.555357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46108)
0927 02:54:59.555429 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.555441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.555480 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.555602 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.555602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.555738 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.555778 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.555779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.556637 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.556638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.557110 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.557111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.557530 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.557531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.557563 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.557936 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.557940 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.559077 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.560722 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.560724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.560724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.560935 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.561079 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.561080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.561080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.561137 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.561318 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.561319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.561364 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.561428 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.561517 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 02:54:59.562968 13711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350: Bootstrap starting.
I20260927 02:54:59.563536 13711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:59.563798 13711 log.cc:824] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:59.564452 13711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350: No bootstrap required, opened a new log
I20260927 02:54:59.565801 13711 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.565908 13711 raft_consensus.cc:374] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:59.565937 13711 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02fc75aaa254406adfafbc628a09350, State: Initialized, Role: FOLLOWER
I20260927 02:54:59.566051 13711 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.566273 13888 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.566325 13711 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:54:59.566349 13888 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:54:59.567865 13899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:54:59.567962 13899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:54:59.569137 13643 master_runner.cc:186] Error getting master registration for 127.0.98.61:44655: OK, instance_id { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" instance_seqno: 1790477699351791 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:54:59.571874 13643 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:54:59.583838 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } attempt: 3
Sep 27 02:54:59 dist-test-slave-mfx9 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:54:59.591686 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.584178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.0.98.60:39773)
0927 02:54:59.584219 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:54:59.584221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:54:59.584232 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:54:59.584254 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:54:59.584254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:54:59.584263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:54:59.584325 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:54:59.584640 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.585378 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.586317 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.586642 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.586705 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.589040 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:54:59.589041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:54:59.589042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:54:59.589049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:54:59.590557 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.590943 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.590944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.590944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.591001 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:54:59.591189 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:54:59.591190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:54:59.591190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:54:59.591313 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:54:59.591374 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:54:59.591523 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:54:59.591594 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:54:59.591688 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:54:59.584105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39773 (local address 127.0.0.1:46110)
0927 02:54:59.584155 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:54:59.584165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:54:59.584201 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:54:59.584335 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:54:59.584336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:54:59.584515 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:54:59.584574 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.584576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.585467 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.585468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.586231 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:54:59.586235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:54:59.586793 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:54:59.586795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:54:59.586857 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:54:59.587427 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:54:59.587431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:54:59.588958 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:54:59.590644 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.590645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.590645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.590869 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.591061 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:54:59.591061 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:54:59.591062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:54:59.591126 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:54:59.591382 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:54:59.591383 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:54:59.591428 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:54:59.591495 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:54:59.591598 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:54:59.593581 13786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0: Bootstrap starting.
I20260927 02:54:59.594307 13786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0: Neither blocks nor log segments found. Creating new log.
I20260927 02:54:59.594635 13786 log.cc:824] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:54:59.595520 13786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0: No bootstrap required, opened a new log
I20260927 02:54:59.597491 13786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.597632 13786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:54:59.597713 13786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8c383c8306426390cc7bb55bd583a0, State: Initialized, Role: FOLLOWER
I20260927 02:54:59.597851 13786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.598147 13786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:54:59.598099 13902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.598817 13902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:54:59.599740 13739 tablet.cc:2369] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:54:59.600906 13913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:54:59.600980 13913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:54:59.603288 13718 master_runner.cc:263] No leader master found from master 9f8c383c8306426390cc7bb55bd583a0
I20260927 02:54:59.603338 13718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:54:59.693616 13902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:54:59.693727 13902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.693938 13902 leader_election.cc:288] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d02fc75aaa254406adfafbc628a09350 (127.0.98.62:34049), b5d20bb702d947c598db2b677f893a78 (127.0.98.60:39773)
I20260927 02:54:59.694346 13685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f8c383c8306426390cc7bb55bd583a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02fc75aaa254406adfafbc628a09350" is_pre_election: true
I20260927 02:54:59.694478 13685 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f8c383c8306426390cc7bb55bd583a0 in term 0.
I20260927 02:54:59.694808 13734 leader_election.cc:302] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f8c383c8306426390cc7bb55bd583a0, d02fc75aaa254406adfafbc628a09350; no voters:
I20260927 02:54:59.694873 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f8c383c8306426390cc7bb55bd583a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d20bb702d947c598db2b677f893a78" is_pre_election: true
I20260927 02:54:59.694978 13902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:54:59.695027 13902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:54:59.695030 13835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f8c383c8306426390cc7bb55bd583a0 in term 0.
I20260927 02:54:59.695060 13902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:59.695907 13902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.696033 13902 leader_election.cc:288] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [CANDIDATE]: Term 1 election: Requested vote from peers d02fc75aaa254406adfafbc628a09350 (127.0.98.62:34049), b5d20bb702d947c598db2b677f893a78 (127.0.98.60:39773)
I20260927 02:54:59.696208 13685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f8c383c8306426390cc7bb55bd583a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02fc75aaa254406adfafbc628a09350"
I20260927 02:54:59.696213 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f8c383c8306426390cc7bb55bd583a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d20bb702d947c598db2b677f893a78"
I20260927 02:54:59.696275 13685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:59.696286 13835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:54:59.697109 13685 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f8c383c8306426390cc7bb55bd583a0 in term 1.
I20260927 02:54:59.697157 13835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f8c383c8306426390cc7bb55bd583a0 in term 1.
I20260927 02:54:59.697304 13734 leader_election.cc:302] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f8c383c8306426390cc7bb55bd583a0, d02fc75aaa254406adfafbc628a09350; no voters:
I20260927 02:54:59.697398 13902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:54:59.697566 13902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [term 1 LEADER]: Becoming Leader. State: Replica: 9f8c383c8306426390cc7bb55bd583a0, State: Running, Role: LEADER
I20260927 02:54:59.697659 13902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } }
I20260927 02:54:59.698064 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8c383c8306426390cc7bb55bd583a0. Latest consensus state: current_term: 1 leader_uuid: "9f8c383c8306426390cc7bb55bd583a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.698159 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:59.698406 13917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:54:59.698626 13917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:54:59.700152 13835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [term 1 FOLLOWER]: Refusing update from remote peer 9f8c383c8306426390cc7bb55bd583a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:54:59.700155 13685 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [term 1 FOLLOWER]: Refusing update from remote peer 9f8c383c8306426390cc7bb55bd583a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:54:59.700474 13915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:54:59.700625 13902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:54:59.701812 13888 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8c383c8306426390cc7bb55bd583a0. Latest consensus state: current_term: 1 leader_uuid: "9f8c383c8306426390cc7bb55bd583a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.701813 13867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8c383c8306426390cc7bb55bd583a0. Latest consensus state: current_term: 1 leader_uuid: "9f8c383c8306426390cc7bb55bd583a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.701889 13888 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:54:59.701890 13867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:54:59.703694 13902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f8c383c8306426390cc7bb55bd583a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.703774 13902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:59.703864 13902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f8c383c8306426390cc7bb55bd583a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.703914 13902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:54:59.704178 13888 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f8c383c8306426390cc7bb55bd583a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.704245 13888 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:54:59.704353 13867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f8c383c8306426390cc7bb55bd583a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02fc75aaa254406adfafbc628a09350" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } peers { permanent_uuid: "9f8c383c8306426390cc7bb55bd583a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44655 } } peers { permanent_uuid: "b5d20bb702d947c598db2b677f893a78" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39773 } } }
I20260927 02:54:59.704429 13867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:54:59.704507 13920 mvcc.cc:205] Tried to move back new op lower bound from 7333796657969664000 to 7333796657961979904. Current Snapshot: MvccSnapshot[applied={T|T < 7333796657969664000}]
I20260927 02:54:59.704710 13918 mvcc.cc:205] Tried to move back new op lower bound from 7333796657969664000 to 7333796657961979904. Current Snapshot: MvccSnapshot[applied={T|T < 7333796657969664000}]
I20260927 02:54:59.704803 13919 mvcc.cc:205] Tried to move back new op lower bound from 7333796657969664000 to 7333796657961979904. Current Snapshot: MvccSnapshot[applied={T|T < 7333796657969664000}]
I20260927 02:54:59.706907 13917 catalog_manager.cc:1347] Generated new cluster ID: 1d182d67abf842e8acdb2c08c1690106
I20260927 02:54:59.706945 13917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:54:59.726651 13917 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:54:59.727643 13917 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:54:59.749804 13917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f8c383c8306426390cc7bb55bd583a0: Generated new TSK 0
I20260927 02:54:59.750108 13917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:55:00.529930 13878 catalog_manager.cc:1259] Loaded cluster ID: 1d182d67abf842e8acdb2c08c1690106
I20260927 02:55:00.529994 13878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78: loading cluster ID for follower catalog manager: success
I20260927 02:55:00.530813 13878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78: acquiring CA information for follower catalog manager: success
I20260927 02:55:00.531109 13878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5d20bb702d947c598db2b677f893a78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:55:00.568387 13899 catalog_manager.cc:1259] Loaded cluster ID: 1d182d67abf842e8acdb2c08c1690106
I20260927 02:55:00.568462 13899 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350: loading cluster ID for follower catalog manager: success
I20260927 02:55:00.569258 13899 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350: acquiring CA information for follower catalog manager: success
I20260927 02:55:00.569561 13899 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d02fc75aaa254406adfafbc628a09350: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:55:00.812103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 13643
I20260927 02:55:00.816708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 13718
I20260927 02:55:00.820662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 13793
2026-09-27T02:55:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2787 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:55:00 dist-test-slave-mfx9 krb5kdc[13932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:55:00 dist-test-slave-mfx9 krb5kdc[13932](info): set up 2 sockets
Sep 27 02:55:00 dist-test-slave-mfx9 krb5kdc[13932](info): commencing operation
krb5kdc: starting...
W20260927 02:55:01.762445 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:55:01 dist-test-slave-mfx9 krb5kdc[13932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:55:01.874193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43449
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:01.950507 13945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:01.950699 13945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:01.950719 13945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:01.951815 13945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:55:01.951854 13945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:01.951867 13945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:01.951879 13945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:55:01.951895 13945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:55:01.953246 13945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:01.953419 13945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:01.953575 13945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:01.958014 13953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:01.958160 13945 server_base.cc:1034] running on GCE node
W20260927 02:55:01.958020 13950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:01.958034 13951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:01.958433 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:01.958663 13945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:01.959798 13945 hybrid_clock.cc:648] HybridClock initialized: now 1790477701959787 us; error 26 us; skew 500 ppm
Sep 27 02:55:01 dist-test-slave-mfx9 krb5kdc[13932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:01.962004 13945 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:55:01.962405 13945 webserver.cc:460] Webserver started at http://127.0.98.62:44389/ using document root <none> and password file <none>
I20260927 02:55:01.962571 13945 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:01.962611 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:01.962695 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:01.963667 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "9a91abec611e466ab00ff569f1d047be"
format_stamp: "Formatted at 2026-09-27 02:55:01 on dist-test-slave-mfx9"
server_key: "4e6ace6fce17bc734e900abb8b96b823"
server_key_iv: "4c6f89014ab0dd784174666181decfbe"
server_key_version: "encryptionkey@0"
I20260927 02:55:01.963932 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a91abec611e466ab00ff569f1d047be"
format_stamp: "Formatted at 2026-09-27 02:55:01 on dist-test-slave-mfx9"
server_key: "4e6ace6fce17bc734e900abb8b96b823"
server_key_iv: "4c6f89014ab0dd784174666181decfbe"
server_key_version: "encryptionkey@0"
I20260927 02:55:01.965009 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:55:01.965605 13960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:01.965744 13945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:55:01.965819 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "9a91abec611e466ab00ff569f1d047be"
format_stamp: "Formatted at 2026-09-27 02:55:01 on dist-test-slave-mfx9"
server_key: "4e6ace6fce17bc734e900abb8b96b823"
server_key_iv: "4c6f89014ab0dd784174666181decfbe"
server_key_version: "encryptionkey@0"
I20260927 02:55:01.965875 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:01.983161 13945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:01.984758 13945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:01.984887 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:01.988503 13945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43449
I20260927 02:55:01.988534 14012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43449 every 8 connection(s)
I20260927 02:55:01.988849 13945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:55:01.989598 14013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:01.991734 14013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be: Bootstrap starting.
I20260927 02:55:01.992238 14013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:01.992476 14013 log.cc:824] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:01.993055 14013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be: No bootstrap required, opened a new log
I20260927 02:55:01.994549 14013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a91abec611e466ab00ff569f1d047be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } }
I20260927 02:55:01.994686 14013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:01.994709 14013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a91abec611e466ab00ff569f1d047be, State: Initialized, Role: FOLLOWER
I20260927 02:55:01.994860 14013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a91abec611e466ab00ff569f1d047be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } }
I20260927 02:55:01.994921 14013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:55:01.994942 14013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:55:01.994987 14013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:01.995445 14013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a91abec611e466ab00ff569f1d047be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } }
I20260927 02:55:01.995528 14013 leader_election.cc:302] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a91abec611e466ab00ff569f1d047be; no voters:
I20260927 02:55:01.995676 14013 leader_election.cc:288] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:55:01.995719 14016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:55:01.995888 14016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [term 1 LEADER]: Becoming Leader. State: Replica: 9a91abec611e466ab00ff569f1d047be, State: Running, Role: LEADER
I20260927 02:55:01.995893 14013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:55:01.996039 14016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a91abec611e466ab00ff569f1d047be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } }
I20260927 02:55:01.996543 14018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a91abec611e466ab00ff569f1d047be. Latest consensus state: current_term: 1 leader_uuid: "9a91abec611e466ab00ff569f1d047be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a91abec611e466ab00ff569f1d047be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } } }
I20260927 02:55:01.996697 14017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a91abec611e466ab00ff569f1d047be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a91abec611e466ab00ff569f1d047be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } } }
I20260927 02:55:01.996847 14017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:01.996763 14018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:01.997066 14028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:55:01.997537 14028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:55:01.997951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 13945
I20260927 02:55:01.998049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:55:01.998224 392 external_mini_cluster.cc:1428] Setting key 6440e445e43d965964ba2091a1bc9209
I20260927 02:55:01.998883 14028 catalog_manager.cc:1347] Generated new cluster ID: 874fcd9205d4414db783726d1931a1ba
I20260927 02:55:01.998936 14028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:55:02 dist-test-slave-mfx9 krb5kdc[13932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:02.008419 14028 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:55:02.008868 14028 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:55:02.009999 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:01.998791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.0.98.62:43449)
0927 02:55:01.999114 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:01.999122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:01.999150 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:55:01.999262 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:01.999268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:01.999319 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:01.999432 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:02.000174 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:02.001525 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:02.002604 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:02.003281 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:02.003378 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:02.006114 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:02.006118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:02.006119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:02.006132 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:02.008118 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:02.008488 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:02.008491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:02.008491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:02.008605 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:02.008793 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:02.008794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:02.008794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:02.009054 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:02.009203 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:02.009617 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 02:55:02.009766 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":62,"threads_started":1}
I20260927 02:55:02.010932 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:55:02.021342 14028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a91abec611e466ab00ff569f1d047be: Generated new TSK 0
I20260927 02:55:02.021513 14028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:55:02.064459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 13945
2026-09-27T02:55:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1243 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:55:02 dist-test-slave-mfx9 krb5kdc[14040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:55:02 dist-test-slave-mfx9 krb5kdc[14040](info): set up 2 sockets
Sep 27 02:55:02 dist-test-slave-mfx9 krb5kdc[14040](info): commencing operation
krb5kdc: starting...
W20260927 02:55:03.002456 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:55:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:55:03.109396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41093
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:03.186280 14053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:03.186461 14053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:03.186481 14053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:03.187590 14053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:55:03.187636 14053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:03.187649 14053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:03.187673 14053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:55:03.187697 14053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:55:03.189098 14053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:03.189308 14053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:03.189498 14053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:55:03.194010 14053 server_base.cc:1034] running on GCE node
W20260927 02:55:03.193964 14058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:03.193994 14061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:03.193964 14059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:03.194325 14053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:03.194562 14053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:03.195710 14053 hybrid_clock.cc:648] HybridClock initialized: now 1790477703195695 us; error 27 us; skew 500 ppm
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:03.198009 14053 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:55:03.198360 14053 webserver.cc:460] Webserver started at http://127.0.98.62:45269/ using document root <none> and password file <none>
I20260927 02:55:03.198523 14053 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:03.198560 14053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:03.198644 14053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:03.199586 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "401f8d880bb34f8f80683113d85950f6"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "9afe4a3979e8698f9c174a2d86239fb5"
server_key_iv: "5567f1db9aac40bd21b6fd709c810d2f"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.199853 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "401f8d880bb34f8f80683113d85950f6"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "9afe4a3979e8698f9c174a2d86239fb5"
server_key_iv: "5567f1db9aac40bd21b6fd709c810d2f"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.200914 14053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:55:03.201476 14068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.201622 14053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:03.201697 14053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "401f8d880bb34f8f80683113d85950f6"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "9afe4a3979e8698f9c174a2d86239fb5"
server_key_iv: "5567f1db9aac40bd21b6fd709c810d2f"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.201747 14053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:03.226464 14053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:03.228097 14053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:03.228197 14053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:03.231613 14053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41093
I20260927 02:55:03.231645 14120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41093 every 8 connection(s)
I20260927 02:55:03.231936 14053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:55:03.232868 14121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:03.232923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 14053
I20260927 02:55:03.233011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:55:03.233213 392 external_mini_cluster.cc:1428] Setting key b0d4601353c243a5b63d6007ac09b59f
I20260927 02:55:03.235843 14121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Bootstrap starting.
I20260927 02:55:03.236452 14121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:03.236744 14121 log.cc:824] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:03.237493 14121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: No bootstrap required, opened a new log
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:03.239763 14121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:03.239914 14121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:03.239946 14121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401f8d880bb34f8f80683113d85950f6, State: Initialized, Role: FOLLOWER
I20260927 02:55:03.240070 14121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:03.240134 14121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:55:03.240157 14121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:55:03.240193 14121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:03.240808 14121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:03.240911 14121 leader_election.cc:302] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401f8d880bb34f8f80683113d85950f6; no voters:
I20260927 02:55:03.241055 14121 leader_election.cc:288] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:55:03.241149 14126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:55:03.241290 14126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 1 LEADER]: Becoming Leader. State: Replica: 401f8d880bb34f8f80683113d85950f6, State: Running, Role: LEADER
I20260927 02:55:03.241302 14121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:55:03.241462 14126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:03.241802 14127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "401f8d880bb34f8f80683113d85950f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } } }
I20260927 02:55:03.241873 14127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:03.242067 14128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401f8d880bb34f8f80683113d85950f6. Latest consensus state: current_term: 1 leader_uuid: "401f8d880bb34f8f80683113d85950f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } } }
I20260927 02:55:03.242285 14128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:03.242427 14133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:55:03.242959 14133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:55:03.243366 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.233698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.0.98.62:41093)
0927 02:55:03.233854 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.233863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.233887 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:55:03.234326 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.234333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.234379 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.234489 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:03.235214 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.236099 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.237090 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.237590 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.237678 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.240051 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:03.240056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:03.240056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:03.240068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:03.241722 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.242113 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.242116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.242117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.242220 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.242413 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.242414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.242415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.242711 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:03.242886 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.243044 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.243184 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
I20260927 02:55:03.244395 14133 catalog_manager.cc:1347] Generated new cluster ID: e31bb802857244fd88a4ac71818bf8ec
I20260927 02:55:03.244441 14133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:55:03.259229 14133 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:55:03.259707 14133 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:55:03.277048 14133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Generated new TSK 0
I20260927 02:55:03.277199 14133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:55:03.303202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41093
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:03.378901 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:03.379068 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:03.379089 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:03.380115 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:03.380146 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:03.380168 14147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:55:03.381495 14147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:03.381687 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:03.381860 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:03.386175 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:03.386251 14147 server_base.cc:1034] running on GCE node
W20260927 02:55:03.386193 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:03.386221 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:03.386523 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:03.386740 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:03.387897 14147 hybrid_clock.cc:648] HybridClock initialized: now 1790477703387885 us; error 25 us; skew 500 ppm
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:03.390076 14147 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:55:03.390431 14147 webserver.cc:460] Webserver started at http://127.0.98.1:40367/ using document root <none> and password file <none>
I20260927 02:55:03.390628 14147 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:03.390673 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:03.390816 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:03.391722 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "92f6eba88737498299c625823b425834"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "c0e9a40520f56c905b65b2c5af8fa717"
server_key_iv: "6ab70b0d619250975a929f3aa908078e"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.392020 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92f6eba88737498299c625823b425834"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "c0e9a40520f56c905b65b2c5af8fa717"
server_key_iv: "6ab70b0d619250975a929f3aa908078e"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.393139 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:55:03.393724 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.393864 14147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:03.393939 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "92f6eba88737498299c625823b425834"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "c0e9a40520f56c905b65b2c5af8fa717"
server_key_iv: "6ab70b0d619250975a929f3aa908078e"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.393996 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:03.417675 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:03.419679 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:03.419806 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:03.419997 14147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:03.420262 14147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:03.420293 14147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.420320 14147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:03.420343 14147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.425575 14147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41303
I20260927 02:55:03.425629 14275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41303 every 8 connection(s)
I20260927 02:55:03.425902 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:55:03.426976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 14147
I20260927 02:55:03.427066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:55:03.427238 392 external_mini_cluster.cc:1428] Setting key eac38e2f0adf46ba714f98ef85a58d3d
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:55:03.434900 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.426528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58829 (local address 127.0.98.62:41093)
0927 02:55:03.426569 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.426571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.426752 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:55:03.427279 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.427281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.427293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.427339 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:03.428352 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.428905 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.429851 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.430144 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.430211 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.432396 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:03.432397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:03.432397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:03.432403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:03.433606 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.434041 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.434042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.434042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.434095 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.434351 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.434352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.434352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.434451 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:03.434498 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.434747 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.434837 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:55:03.435071 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.426504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:58829)
0927 02:55:03.426644 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.426676 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:03.427184 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:03.427519 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:03.427524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:03.428078 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:03.428262 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.428266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.428989 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.428990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.429764 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.429766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.430214 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.430215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.430310 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.430725 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:03.430733 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:03.432306 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:03.433690 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:03.433697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:03.433699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:03.433963 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:03.434166 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:03.434167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:03.434167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:03.434294 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:03.434508 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:03.434513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:03.434649 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 02:55:03.434731 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:55:03.434877 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260927 02:55:03.435549 14276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41093
I20260927 02:55:03.435622 14276 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:03.435753 14276 heartbeater.cc:511] Master 127.0.98.62:41093 requested a full tablet report, sending...
I20260927 02:55:03.436169 14085 ts_manager.cc:194] Registered new tserver with Master: 92f6eba88737498299c625823b425834 (127.0.98.1:41303)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:55:03.436780 14085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:55:03.449774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41093
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:03.525434 14281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:03.525620 14281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:03.525650 14281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:03.526770 14281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:03.526806 14281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:03.526840 14281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:55:03.528223 14281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:03.528430 14281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:03.528623 14281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:03.533116 14289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:03.533105 14287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:03.533107 14286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:03.533282 14281 server_base.cc:1034] running on GCE node
I20260927 02:55:03.533463 14281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:03.533676 14281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:03.534811 14281 hybrid_clock.cc:648] HybridClock initialized: now 1790477703534795 us; error 28 us; skew 500 ppm
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:03.537014 14281 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:55:03.537400 14281 webserver.cc:460] Webserver started at http://127.0.98.2:45647/ using document root <none> and password file <none>
I20260927 02:55:03.537570 14281 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:03.537607 14281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:03.537698 14281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:03.538610 14281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "af9d583b24b74ceea0564a13a95713f0"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "cf8fd670b52737d07499a11093362915"
server_key_iv: "7ba1587d85eca454cbd5c8d741435505"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.538938 14281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af9d583b24b74ceea0564a13a95713f0"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "cf8fd670b52737d07499a11093362915"
server_key_iv: "7ba1587d85eca454cbd5c8d741435505"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.540040 14281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:03.540630 14296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.540813 14281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:03.540882 14281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "af9d583b24b74ceea0564a13a95713f0"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "cf8fd670b52737d07499a11093362915"
server_key_iv: "7ba1587d85eca454cbd5c8d741435505"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.540944 14281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:03.560286 14281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:03.562130 14281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:03.562244 14281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:03.562458 14281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:03.562731 14281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:03.562786 14281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.562822 14281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:03.562847 14281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.567996 14281 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34203
I20260927 02:55:03.568113 14409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34203 every 8 connection(s)
I20260927 02:55:03.568351 14281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:55:03.573632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 14281
I20260927 02:55:03.573745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:55:03.573935 392 external_mini_cluster.cc:1428] Setting key e5a5fc5a9f0d1dfa5eb38b3ab91c033f
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:03.577013 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.568951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53827 (local address 127.0.98.62:41093)
0927 02:55:03.569001 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.569002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.569184 (+ 182us) server_negotiation.cc:407] Connection header received
0927 02:55:03.569687 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.569688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.569699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.569750 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:03.570642 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.571172 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.572117 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.572308 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.572364 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.574557 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:03.574558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:03.574559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:03.574566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:03.575779 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.576143 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.576144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.576144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.576200 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.576406 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.576407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.576407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.576519 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:03.576573 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.576831 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.576911 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:55:03.577186 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.568924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:53827)
0927 02:55:03.569075 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.569114 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:03.569583 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:03.569810 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:03.569813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:03.570358 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:03.570544 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.570549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.571255 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.571255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.572038 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.572040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.572383 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.572384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.572493 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.572957 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:03.572965 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:03.574458 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:03.575854 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:03.575860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:03.575862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:03.576075 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:03.576252 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:03.576253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:03.576253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:03.576352 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:03.576575 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:03.576581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:03.576710 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 02:55:03.576830 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 02:55:03.576982 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260927 02:55:03.577644 14410 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41093
I20260927 02:55:03.577744 14410 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:03.577925 14410 heartbeater.cc:511] Master 127.0.98.62:41093 requested a full tablet report, sending...
I20260927 02:55:03.578353 14085 ts_manager.cc:194] Registered new tserver with Master: af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203)
I20260927 02:55:03.578868 14085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53827
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:55:03.595268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41093
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:03.671586 14415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:03.671777 14415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:03.671797 14415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:03.672881 14415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:03.672919 14415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:03.672945 14415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:55:03.674307 14415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:03.674511 14415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:03.674706 14415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:03.679229 14421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:03.679261 14420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:03.679291 14415 server_base.cc:1034] running on GCE node
W20260927 02:55:03.679250 14423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:03.679553 14415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:03.679761 14415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:03.680891 14415 hybrid_clock.cc:648] HybridClock initialized: now 1790477703680879 us; error 26 us; skew 500 ppm
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:03.683152 14415 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:55:03.683533 14415 webserver.cc:460] Webserver started at http://127.0.98.3:39705/ using document root <none> and password file <none>
I20260927 02:55:03.683733 14415 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:03.683779 14415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:03.683892 14415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:03.684803 14415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d117946453444bd80b8039c60e49cf7"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "146435a53bc103484221f4a2b6cd9ac3"
server_key_iv: "9c87d8c60fd6574902e2210eea1a2354"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.685107 14415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d117946453444bd80b8039c60e49cf7"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "146435a53bc103484221f4a2b6cd9ac3"
server_key_iv: "9c87d8c60fd6574902e2210eea1a2354"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.686192 14415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:55:03.686748 14430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.686918 14415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.686992 14415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "6d117946453444bd80b8039c60e49cf7"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "146435a53bc103484221f4a2b6cd9ac3"
server_key_iv: "9c87d8c60fd6574902e2210eea1a2354"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.687053 14415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:03.718436 14415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:03.720572 14415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:03.720726 14415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:03.720937 14415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:03.721217 14415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:03.721249 14415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.721279 14415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:03.721307 14415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.726673 14415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37119
I20260927 02:55:03.726714 14543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37119 every 8 connection(s)
I20260927 02:55:03.727017 14415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:55:03.729027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 14415
I20260927 02:55:03.729107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:55:03.729260 392 external_mini_cluster.cc:1428] Setting key 3e4e1f8f11eb2962680bde889ce7b0e9
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:03.735234 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.727614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41917 (local address 127.0.98.62:41093)
0927 02:55:03.727670 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.727672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.727838 (+ 166us) server_negotiation.cc:407] Connection header received
0927 02:55:03.728334 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.728335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.728348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.728400 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:03.729388 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.730028 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.731035 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.731240 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.731293 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.733055 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:03.733056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:03.733056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:03.733063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:03.734056 (+ 993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.734382 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.734383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.734384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.734436 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.734634 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.734635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.734635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.734741 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:03.734832 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.735068 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.735149 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:55:03.735378 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.727585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:41917)
0927 02:55:03.727720 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.727763 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:03.728236 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:03.728469 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:03.728473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:03.729061 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:03.729306 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.729313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.730103 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.730104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.730962 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.730964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.731304 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.731305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.731427 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.731767 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:03.731775 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:03.732969 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:03.734122 (+ 1153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:03.734128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:03.734130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:03.734321 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:03.734487 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:03.734488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:03.734488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:03.734583 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:03.734825 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:03.734830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:03.734956 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:55:03.735042 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:55:03.735179 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260927 02:55:03.735761 14544 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41093
I20260927 02:55:03.735833 14544 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:03.735961 14544 heartbeater.cc:511] Master 127.0.98.62:41093 requested a full tablet report, sending...
I20260927 02:55:03.736274 14085 ts_manager.cc:194] Registered new tserver with Master: 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119)
I20260927 02:55:03.736629 14085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41917
I20260927 02:55:03.740725 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:55:03.747663 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.742048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.98.62:41093)
0927 02:55:03.742090 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.742092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.742298 (+ 206us) server_negotiation.cc:407] Connection header received
0927 02:55:03.742400 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.742400 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.742409 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.742461 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:03.743140 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.743704 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.744424 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.744625 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.744663 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.745629 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:03.745630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:03.745631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:03.745635 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:03.746819 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.747133 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.747133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.747133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.747175 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:03.747292 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:03.747293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:03.747293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:03.747409 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:03.747475 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.747552 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.747604 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:55:03.749051 14085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:55:03.750504 14085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:55:03.756165 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.752529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.0.98.1:41303)
0927 02:55:03.752772 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.752777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.752786 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:55:03.752827 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.752829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.752844 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.752921 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:03.753077 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.753859 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.754992 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.755725 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.755789 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.755840 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.756035 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.756076 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
I20260927 02:55:03.756258 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.752358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.0.1:46994)
0927 02:55:03.752586 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.752636 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:03.752698 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:03.752926 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:03.752929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:03.752931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:03.752991 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.752992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.753947 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.753947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.754911 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.754913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.755870 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.755871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.755940 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.755954 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:55:03.756009 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:55:03.756192 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":53,"threads_started":1}
I20260927 02:55:03.756456 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.752358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37119 (local address 127.0.0.1:41666)
0927 02:55:03.752683 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.752693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:03.752743 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:03.753202 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:03.753203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:03.753205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:03.753249 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.753250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.754503 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.754504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.755453 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.755455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.756142 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.756143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.756288 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.756293 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:55:03.756340 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:55:03.756380 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":4,"spinlock_wait_cycles":1280,"thread_start_us":48,"threads_started":1}
I20260927 02:55:03.756735 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.752508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.0.98.3:37119)
0927 02:55:03.753049 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.753053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.753061 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:55:03.753092 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.753094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.753108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.753316 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:03.753374 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.754398 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.755521 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.755995 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.756058 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.756109 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.756621 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.756660 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":72,"threads_started":1}
I20260927 02:55:03.756912 14210 tablet_service.cc:1467] Processing CreateTablet for tablet 15aa98edf733431199faadc11a854e41 (DEFAULT_TABLE table=test-table [id=5fbdc9eb045247f89ab649612cf606ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:03.757200 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.752585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.0.98.2:34203)
0927 02:55:03.752733 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.752738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.752849 (+ 111us) server_negotiation.cc:407] Connection header received
0927 02:55:03.752927 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.752929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.752943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.753019 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:03.753233 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.754399 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.755994 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.756869 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.756931 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.756986 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.757076 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.757124 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260927 02:55:03.757213 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.752368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34203 (local address 127.0.0.1:55310)
0927 02:55:03.752804 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.752811 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:03.752854 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:03.753025 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:03.753026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:03.753028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:03.753154 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.753156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.754820 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.754821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.755917 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.755918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.756951 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.756952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.757013 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.757017 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:55:03.757066 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:55:03.757152 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":44,"threads_started":1}
I20260927 02:55:03.757287 14210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15aa98edf733431199faadc11a854e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:03.757328 14478 tablet_service.cc:1467] Processing CreateTablet for tablet 15aa98edf733431199faadc11a854e41 (DEFAULT_TABLE table=test-table [id=5fbdc9eb045247f89ab649612cf606ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:03.757628 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15aa98edf733431199faadc11a854e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:03.758028 14344 tablet_service.cc:1467] Processing CreateTablet for tablet 15aa98edf733431199faadc11a854e41 (DEFAULT_TABLE table=test-table [id=5fbdc9eb045247f89ab649612cf606ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:03.758327 14344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15aa98edf733431199faadc11a854e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:03.759824 14558 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Bootstrap starting.
I20260927 02:55:03.760409 14558 tablet_bootstrap.cc:654] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:03.760658 14559 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: Bootstrap starting.
I20260927 02:55:03.760949 14558 log.cc:824] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:03.761423 14559 tablet_bootstrap.cc:654] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:03.761409 14561 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Bootstrap starting.
I20260927 02:55:03.761679 14558 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: No bootstrap required, opened a new log
I20260927 02:55:03.761739 14558 ts_tablet_manager.cc:1397] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:55:03.762010 14561 tablet_bootstrap.cc:654] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:03.762015 14559 log.cc:824] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:03.762454 14561 log.cc:824] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:03.762724 14559 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: No bootstrap required, opened a new log
I20260927 02:55:03.762804 14559 ts_tablet_manager.cc:1397] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:55:03.763162 14561 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: No bootstrap required, opened a new log
I20260927 02:55:03.763227 14561 ts_tablet_manager.cc:1397] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:55:03.763409 14558 raft_consensus.cc:348] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.763540 14558 raft_consensus.cc:374] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:03.763597 14558 raft_consensus.cc:729] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f6eba88737498299c625823b425834, State: Initialized, Role: FOLLOWER
I20260927 02:55:03.763715 14558 consensus_queue.cc:260] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.763962 14558 ts_tablet_manager.cc:1428] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:55:03.764122 14276 heartbeater.cc:503] Master 127.0.98.62:41093 was elected leader, sending a full tablet report...
I20260927 02:55:03.764582 14559 raft_consensus.cc:348] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.764745 14559 raft_consensus.cc:374] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:03.764780 14559 raft_consensus.cc:729] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d117946453444bd80b8039c60e49cf7, State: Initialized, Role: FOLLOWER
I20260927 02:55:03.764875 14559 consensus_queue.cc:260] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.765002 14561 raft_consensus.cc:348] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.765123 14559 ts_tablet_manager.cc:1428] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:55:03.765143 14544 heartbeater.cc:503] Master 127.0.98.62:41093 was elected leader, sending a full tablet report...
I20260927 02:55:03.765153 14561 raft_consensus.cc:374] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:03.765189 14561 raft_consensus.cc:729] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9d583b24b74ceea0564a13a95713f0, State: Initialized, Role: FOLLOWER
I20260927 02:55:03.765312 14561 consensus_queue.cc:260] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.765568 14410 heartbeater.cc:503] Master 127.0.98.62:41093 was elected leader, sending a full tablet report...
I20260927 02:55:03.765566 14561 ts_tablet_manager.cc:1428] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260927 02:55:03.818879 14411 tablet.cc:2369] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:55:03.839350 14565 raft_consensus.cc:482] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:03.839457 14565 raft_consensus.cc:504] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.839744 14565 leader_election.cc:288] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f6eba88737498299c625823b425834 (127.0.98.1:41303), af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203)
I20260927 02:55:03.842566 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.839983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47457 (local address 127.0.98.2:34203)
0927 02:55:03.840027 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.840029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.840038 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:55:03.840074 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.840074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.840085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.840148 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:03.840288 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.840807 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.841782 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.842244 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.842303 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.842313 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.842469 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.842501 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:55:03.842572 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.839953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41237 (local address 127.0.98.1:41303)
0927 02:55:03.839993 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.839995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.840130 (+ 135us) server_negotiation.cc:407] Connection header received
0927 02:55:03.840171 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.840172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.840183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.840227 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:03.840358 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.840862 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.841859 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.842269 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.842326 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.842336 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.842459 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.842509 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:55:03.842573 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.839894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34203 (local address 127.0.98.3:47457)
0927 02:55:03.839963 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.839976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:03.840029 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:03.840162 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:03.840163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:03.840165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:03.840231 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.840232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.840864 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.840865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.841708 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.841711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.842313 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.842315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.842395 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.842398 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:55:03.842441 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:55:03.842481 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":3}
I20260927 02:55:03.842594 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.839904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.98.3:41237)
0927 02:55:03.840062 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.840071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:03.840117 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:03.840235 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:03.840236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:03.840237 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:03.840304 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.840305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.840933 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.840934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.841780 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:03.841782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.842336 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.842337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:03.842402 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.842405 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:55:03.842448 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:55:03.842484 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":12,"thread_start_us":98,"threads_started":1}
I20260927 02:55:03.842959 14364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15aa98edf733431199faadc11a854e41" candidate_uuid: "6d117946453444bd80b8039c60e49cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9d583b24b74ceea0564a13a95713f0" is_pre_election: true
I20260927 02:55:03.843114 14364 raft_consensus.cc:2454] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d117946453444bd80b8039c60e49cf7 in term 0.
I20260927 02:55:03.843133 14230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15aa98edf733431199faadc11a854e41" candidate_uuid: "6d117946453444bd80b8039c60e49cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f6eba88737498299c625823b425834" is_pre_election: true
I20260927 02:55:03.843261 14230 raft_consensus.cc:2454] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d117946453444bd80b8039c60e49cf7 in term 0.
I20260927 02:55:03.843327 14431 leader_election.cc:302] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d117946453444bd80b8039c60e49cf7, af9d583b24b74ceea0564a13a95713f0; no voters:
I20260927 02:55:03.843464 14565 raft_consensus.cc:2781] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:55:03.843519 14565 raft_consensus.cc:482] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:55:03.843540 14565 raft_consensus.cc:3037] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:03.844197 14565 raft_consensus.cc:504] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.844314 14565 leader_election.cc:288] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [CANDIDATE]: Term 1 election: Requested vote from peers 92f6eba88737498299c625823b425834 (127.0.98.1:41303), af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203)
I20260927 02:55:03.844446 14230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15aa98edf733431199faadc11a854e41" candidate_uuid: "6d117946453444bd80b8039c60e49cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f6eba88737498299c625823b425834"
I20260927 02:55:03.844481 14364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15aa98edf733431199faadc11a854e41" candidate_uuid: "6d117946453444bd80b8039c60e49cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9d583b24b74ceea0564a13a95713f0"
I20260927 02:55:03.844521 14230 raft_consensus.cc:3037] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:03.844566 14364 raft_consensus.cc:3037] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:03.845356 14230 raft_consensus.cc:2454] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d117946453444bd80b8039c60e49cf7 in term 1.
I20260927 02:55:03.845356 14364 raft_consensus.cc:2454] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d117946453444bd80b8039c60e49cf7 in term 1.
I20260927 02:55:03.845515 14433 leader_election.cc:302] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d117946453444bd80b8039c60e49cf7, 92f6eba88737498299c625823b425834; no voters:
I20260927 02:55:03.845583 14565 raft_consensus.cc:2781] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:55:03.845708 14565 raft_consensus.cc:686] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 1 LEADER]: Becoming Leader. State: Replica: 6d117946453444bd80b8039c60e49cf7, State: Running, Role: LEADER
I20260927 02:55:03.845786 14565 consensus_queue.cc:237] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:03.846565 14085 catalog_manager.cc:5495] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d117946453444bd80b8039c60e49cf7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6d117946453444bd80b8039c60e49cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } health_report { overall_health: UNKNOWN } } }
I20260927 02:55:03.872998 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.871026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.0.98.3:37119)
0927 02:55:03.871080 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:03.871081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:03.871102 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:55:03.871135 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:03.871136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:03.871153 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:03.871202 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:55:03.871290 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.871741 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.872428 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:03.872609 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:03.872652 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:03.872756 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:55:03.872846 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:55:03.872898 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:03.872914 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 02:55:03.872929 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:55:03.874696 14230 raft_consensus.cc:1261] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Refusing update from remote peer 6d117946453444bd80b8039c60e49cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:55:03.874696 14364 raft_consensus.cc:1261] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 1 FOLLOWER]: Refusing update from remote peer 6d117946453444bd80b8039c60e49cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:55:03.875063 14565 consensus_queue.cc:1035] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:55:03.875201 14568 consensus_queue.cc:1035] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:55:03.877960 14571 mvcc.cc:205] Tried to move back new op lower bound from 7333796675068190720 to 7333796674953052160. Current Snapshot: MvccSnapshot[applied={T|T < 7333796675068190720}]
I20260927 02:55:03.878347 14572 mvcc.cc:205] Tried to move back new op lower bound from 7333796675068190720 to 7333796674953052160. Current Snapshot: MvccSnapshot[applied={T|T < 7333796675068190720}]
I20260927 02:55:03.879956 14085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:55:03.880048 14085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:55:03.880730 14085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Sending DeleteTablet for 3 replicas of tablet 15aa98edf733431199faadc11a854e41
I20260927 02:55:03.881048 14478 tablet_service.cc:1514] Processing DeleteTablet for tablet 15aa98edf733431199faadc11a854e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41666
I20260927 02:55:03.881067 14210 tablet_service.cc:1514] Processing DeleteTablet for tablet 15aa98edf733431199faadc11a854e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46994
I20260927 02:55:03.881258 14344 tablet_service.cc:1514] Processing DeleteTablet for tablet 15aa98edf733431199faadc11a854e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55310
I20260927 02:55:03.881311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 14147
I20260927 02:55:03.881816 14582 tablet_replica.cc:330] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: stopping tablet replica
I20260927 02:55:03.881835 14580 tablet_replica.cc:330] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: stopping tablet replica
I20260927 02:55:03.881915 14580 raft_consensus.cc:2229] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:55:03.882001 14582 raft_consensus.cc:2229] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:55:03.882027 14580 raft_consensus.cc:2258] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:03.882149 14582 raft_consensus.cc:2258] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:03.882565 14580 ts_tablet_manager.cc:1905] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:03.882565 14582 ts_tablet_manager.cc:1905] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:03.884207 14580 ts_tablet_manager.cc:1918] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:03.884217 14582 ts_tablet_manager.cc:1918] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:03.884277 14580 log.cc:1196] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/15aa98edf733431199faadc11a854e41
I20260927 02:55:03.884375 14582 log.cc:1196] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/15aa98edf733431199faadc11a854e41
I20260927 02:55:03.884521 14580 ts_tablet_manager.cc:1939] T 15aa98edf733431199faadc11a854e41 P 6d117946453444bd80b8039c60e49cf7: Deleting consensus metadata
I20260927 02:55:03.884722 14582 ts_tablet_manager.cc:1939] T 15aa98edf733431199faadc11a854e41 P af9d583b24b74ceea0564a13a95713f0: Deleting consensus metadata
I20260927 02:55:03.884892 14070 catalog_manager.cc:4841] TS 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119): tablet 15aa98edf733431199faadc11a854e41 (table test-table [id=5fbdc9eb045247f89ab649612cf606ef]) successfully deleted
I20260927 02:55:03.885169 14069 catalog_manager.cc:4841] TS af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203): tablet 15aa98edf733431199faadc11a854e41 (table test-table [id=5fbdc9eb045247f89ab649612cf606ef]) successfully deleted
W20260927 02:55:03.886603 14071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41303 (error 108)
I20260927 02:55:03.886749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 14281
I20260927 02:55:03.891467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 14415
I20260927 02:55:03.891496 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:03.891360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.0.1:47008)
0927 02:55:03.891418 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:03.891465 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:55:03.891597 14071 catalog_manager.cc:4584] TS 92f6eba88737498299c625823b425834 (127.0.98.1:41303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15aa98edf733431199faadc11a854e41: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41303: connect: Connection refused (error 111)
I20260927 02:55:03.896104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 14053
I20260927 02:55:03.900437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41093
--webserver_interface=127.0.98.62
--webserver_port=45269
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:03.976537 14584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:03.976724 14584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:03.976743 14584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:03.977803 14584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:55:03.977842 14584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:03.977855 14584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:03.977866 14584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:55:03.977876 14584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:55:03.979277 14584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:03.979450 14584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:03.979605 14584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:03.987201 14591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:03.987252 14590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:03.987365 14584 server_base.cc:1034] running on GCE node
W20260927 02:55:03.987224 14593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:03.987670 14584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:03.987946 14584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:03.989068 14584 hybrid_clock.cc:648] HybridClock initialized: now 1790477703989054 us; error 32 us; skew 500 ppm
Sep 27 02:55:03 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:03.991293 14584 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:55:03.991667 14584 webserver.cc:460] Webserver started at http://127.0.98.62:45269/ using document root <none> and password file <none>
I20260927 02:55:03.991863 14584 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:03.991919 14584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:03.993131 14584 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:03.993660 14600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.993800 14584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:03.993891 14584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "401f8d880bb34f8f80683113d85950f6"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "9afe4a3979e8698f9c174a2d86239fb5"
server_key_iv: "5567f1db9aac40bd21b6fd709c810d2f"
server_key_version: "encryptionkey@0"
I20260927 02:55:03.994246 14584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:04.014744 14584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:04.016386 14584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:04.016505 14584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:04.020072 14584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41093
I20260927 02:55:04.020117 14652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41093 every 8 connection(s)
I20260927 02:55:04.020453 14584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:55:04.021921 14653 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:55:04.022481 14653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Bootstrap starting.
I20260927 02:55:04.024281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 14584
I20260927 02:55:04.024485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41303
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41093
--webserver_port=40367
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:55:04.030428 14653 log.cc:824] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:04.032194 14653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:55:04.032428 14653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Bootstrap complete.
I20260927 02:55:04.034435 14653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:04.034649 14653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401f8d880bb34f8f80683113d85950f6, State: Initialized, Role: FOLLOWER
I20260927 02:55:04.034807 14653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:04.034873 14653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:55:04.034905 14653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:55:04.034950 14653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:55:04.035876 14653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:04.035957 14653 leader_election.cc:302] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401f8d880bb34f8f80683113d85950f6; no voters:
I20260927 02:55:04.036098 14653 leader_election.cc:288] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:55:04.036159 14657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:55:04.036275 14657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 2 LEADER]: Becoming Leader. State: Replica: 401f8d880bb34f8f80683113d85950f6, State: Running, Role: LEADER
I20260927 02:55:04.036347 14653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:55:04.036355 14657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:04.036675 14657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401f8d880bb34f8f80683113d85950f6. Latest consensus state: current_term: 2 leader_uuid: "401f8d880bb34f8f80683113d85950f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } } }
I20260927 02:55:04.036756 14657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:04.036679 14659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "401f8d880bb34f8f80683113d85950f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } } }
I20260927 02:55:04.036904 14659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:04.036948 14663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:55:04.037725 14663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:55:04.037837 14663 catalog_manager.cc:1259] Loaded cluster ID: e31bb802857244fd88a4ac71818bf8ec
I20260927 02:55:04.038017 14663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:55:04.038269 14663 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 02:55:04.041010 14663 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 02:55:04.043342 14663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:55:04.043517 14663 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Loaded TSK: 0
I20260927 02:55:04.043715 14663 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:55:04.131745 14655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:04.131992 14655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:04.132028 14655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:04.133620 14655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:04.133690 14655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:04.133728 14655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:55:04.135783 14655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40367
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:04.136014 14655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:04.136258 14655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:04.141521 14680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.141535 14681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.141530 14683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:04.141851 14655 server_base.cc:1034] running on GCE node
I20260927 02:55:04.142033 14655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:04.142279 14655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:04.143452 14655 hybrid_clock.cc:648] HybridClock initialized: now 1790477704143434 us; error 34 us; skew 500 ppm
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:04.146413 14655 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:55:04.146872 14655 webserver.cc:460] Webserver started at http://127.0.98.1:40367/ using document root <none> and password file <none>
I20260927 02:55:04.147135 14655 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:04.147212 14655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:04.148540 14655 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:55:04.149222 14690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.149403 14655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:55:04.149495 14655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "92f6eba88737498299c625823b425834"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "c0e9a40520f56c905b65b2c5af8fa717"
server_key_iv: "6ab70b0d619250975a929f3aa908078e"
server_key_version: "encryptionkey@0"
I20260927 02:55:04.149802 14655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:04.169623 14655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:04.171844 14655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:04.171998 14655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:04.172225 14655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:04.172729 14697 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:55:04.174522 14655 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:55:04.174575 14655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:55:04.174614 14655 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:55:04.175246 14655 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:55:04.175282 14655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:55:04.175361 14697 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Bootstrap starting.
I20260927 02:55:04.182389 14655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41303
I20260927 02:55:04.182777 14655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:55:04.185424 14697 log.cc:824] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:04.187186 14804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41303 every 8 connection(s)
I20260927 02:55:04.187733 14697 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:55:04.187968 14697 tablet_bootstrap.cc:492] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Bootstrap complete.
I20260927 02:55:04.188687 14697 ts_tablet_manager.cc:1397] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260927 02:55:04.189703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 14655
I20260927 02:55:04.189988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34203
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41093
--webserver_port=45647
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:55:04.189942 14697 raft_consensus.cc:348] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:04.190255 14697 raft_consensus.cc:729] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f6eba88737498299c625823b425834, State: Initialized, Role: FOLLOWER
I20260927 02:55:04.190404 14697 consensus_queue.cc:260] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } }
I20260927 02:55:04.190639 14697 ts_tablet_manager.cc:1428] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:04.202070 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.183467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:37013)
0927 02:55:04.185677 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.185719 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.186274 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.186904 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.186909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.187252 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:04.187531 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.187536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.189129 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.189130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.190749 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.190751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.191443 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.191444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.191572 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.197540 (+ 5968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:04.197548 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:04.198976 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:04.200597 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:04.200603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:04.200605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:04.200829 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:04.201063 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:04.201064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:04.201064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:04.201163 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:04.201487 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:04.201493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:04.201617 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.201702 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.201865 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2170,"thread_start_us":97,"threads_started":1}
I20260927 02:55:04.202068 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.183556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37013 (local address 127.0.98.62:41093)
0927 02:55:04.186469 (+ 2913us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.186478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.186512 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:55:04.186653 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.186658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.186710 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.186850 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:04.187665 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.189030 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.190839 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.191248 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.191351 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.199076 (+ 7725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:04.199080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:04.199080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:04.199089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:04.200519 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.200907 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.200910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.200910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.201004 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.201216 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.201217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.201217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.201423 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:04.201489 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.201758 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.201879 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2867,"thread_start_us":72,"threads_started":1}
I20260927 02:55:04.202647 14805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41093
I20260927 02:55:04.202726 14805 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:04.202903 14805 heartbeater.cc:511] Master 127.0.98.62:41093 requested a full tablet report, sending...
I20260927 02:55:04.203537 14617 ts_manager.cc:194] Registered new tserver with Master: 92f6eba88737498299c625823b425834 (127.0.98.1:41303)
I20260927 02:55:04.204563 14617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37013
I20260927 02:55:04.205413 14805 heartbeater.cc:503] Master 127.0.98.62:41093 was elected leader, sending a full tablet report...
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:55:04.211843 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.204327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.0.98.1:41303)
0927 02:55:04.204477 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.204481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.204490 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:55:04.204521 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.204522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.204536 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.204605 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:04.205126 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.206055 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.207072 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.207338 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.207388 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.209210 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:04.209217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:04.209217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:04.209224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:04.210746 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.211148 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.211151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.211151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.211217 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.211384 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.211384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.211384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.211499 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:04.211558 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.211714 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.211783 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260927 02:55:04.211887 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.204080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.0.1:47010)
0927 02:55:04.204255 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.204297 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.204366 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.204618 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.204692 (+ 74us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.204996 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:04.205050 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.205053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.206115 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.206116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.207004 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.207007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.207397 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.207400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.207472 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.207840 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:04.207844 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:04.209136 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:04.210850 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:04.210855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:04.210857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:04.211082 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:04.211275 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:04.211275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:04.211275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:04.211334 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:04.211552 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:04.211557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:04.211608 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.211691 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.211794 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260927 02:55:04.212247 14739 tablet_service.cc:1514] Processing DeleteTablet for tablet 15aa98edf733431199faadc11a854e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47010
I20260927 02:55:04.212702 14814 tablet_replica.cc:330] stopping tablet replica
I20260927 02:55:04.212802 14814 raft_consensus.cc:2229] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:55:04.212864 14814 raft_consensus.cc:2258] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:04.213151 14814 ts_tablet_manager.cc:1905] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:04.215246 14814 ts_tablet_manager.cc:1918] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:04.215309 14814 log.cc:1196] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/15aa98edf733431199faadc11a854e41
I20260927 02:55:04.215498 14814 ts_tablet_manager.cc:1939] T 15aa98edf733431199faadc11a854e41 P 92f6eba88737498299c625823b425834: Deleting consensus metadata
I20260927 02:55:04.215992 14603 catalog_manager.cc:4841] TS 92f6eba88737498299c625823b425834 (127.0.98.1:41303): tablet 15aa98edf733431199faadc11a854e41 (table test-table [id=5fbdc9eb045247f89ab649612cf606ef]) successfully deleted
W20260927 02:55:04.273700 14811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:04.273873 14811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:04.273893 14811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:04.274952 14811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:04.274984 14811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:04.275007 14811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:55:04.276332 14811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45647
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:04.276497 14811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:04.276657 14811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:04.281136 14819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.281173 14820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:04.281268 14811 server_base.cc:1034] running on GCE node
W20260927 02:55:04.281174 14822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:04.281529 14811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:04.281739 14811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:04.282862 14811 hybrid_clock.cc:648] HybridClock initialized: now 1790477704282848 us; error 24 us; skew 500 ppm
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:04.285068 14811 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:55:04.285418 14811 webserver.cc:460] Webserver started at http://127.0.98.2:45647/ using document root <none> and password file <none>
I20260927 02:55:04.285588 14811 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:04.285634 14811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:04.286862 14811 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:04.287376 14829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.287508 14811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:04.287578 14811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "af9d583b24b74ceea0564a13a95713f0"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "cf8fd670b52737d07499a11093362915"
server_key_iv: "7ba1587d85eca454cbd5c8d741435505"
server_key_version: "encryptionkey@0"
I20260927 02:55:04.287851 14811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:04.299216 14811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:04.301111 14811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:04.301215 14811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:04.301371 14811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:04.301599 14811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:04.301625 14811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.301645 14811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:04.301661 14811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.306823 14811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34203
I20260927 02:55:04.306876 14942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34203 every 8 connection(s)
I20260927 02:55:04.307152 14811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:04.314713 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.307731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48415 (local address 127.0.98.62:41093)
0927 02:55:04.307780 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.307781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.307972 (+ 191us) server_negotiation.cc:407] Connection header received
0927 02:55:04.308415 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.308416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.308428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.308474 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:04.309301 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.309773 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.310675 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.310908 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.310953 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.312726 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:04.312727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:04.312727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:04.312734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:04.313735 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.314041 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.314042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.314042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.314085 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.314265 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.314265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.314265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.314354 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:04.314398 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.314580 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.314650 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:55:04.314872 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.307706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:48415)
0927 02:55:04.307847 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.307878 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.308338 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.308524 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.308528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.309042 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:04.309234 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.309238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.309830 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.309831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.310614 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.310617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.310963 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.310964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.311098 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.311462 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:04.311469 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:04.312648 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:04.313794 (+ 1146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:04.313799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:04.313801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:04.313987 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:04.314128 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:04.314128 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:04.314129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:04.314222 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:04.314395 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:04.314398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:04.314494 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.314564 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.314672 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260927 02:55:04.315194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 14811
I20260927 02:55:04.315248 14943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41093
I20260927 02:55:04.315335 14943 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:04.315428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37119
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41093
--webserver_port=39705
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:55:04.315496 14943 heartbeater.cc:511] Master 127.0.98.62:41093 requested a full tablet report, sending...
I20260927 02:55:04.315835 14617 ts_manager.cc:194] Registered new tserver with Master: af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203)
I20260927 02:55:04.316185 14617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48415
W20260927 02:55:04.391994 14946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:04.392200 14946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:04.392230 14946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:04.393307 14946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:04.393344 14946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:04.393378 14946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:55:04.394731 14946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39705
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:04.394959 14946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:04.395147 14946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:04.399662 14952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.399682 14951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.399705 14954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:04.399763 14946 server_base.cc:1034] running on GCE node
I20260927 02:55:04.399984 14946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:04.400192 14946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:04.401331 14946 hybrid_clock.cc:648] HybridClock initialized: now 1790477704401302 us; error 39 us; skew 500 ppm
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:04.403580 14946 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:55:04.403960 14946 webserver.cc:460] Webserver started at http://127.0.98.3:39705/ using document root <none> and password file <none>
I20260927 02:55:04.404145 14946 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:04.404211 14946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:04.405375 14946 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:04.405821 14961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.405957 14946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.406023 14946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "6d117946453444bd80b8039c60e49cf7"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "146435a53bc103484221f4a2b6cd9ac3"
server_key_iv: "9c87d8c60fd6574902e2210eea1a2354"
server_key_version: "encryptionkey@0"
I20260927 02:55:04.406311 14946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:04.424520 14946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:04.426484 14946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:04.426610 14946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:04.426853 14946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:04.427124 14946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:04.427165 14946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.427193 14946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:04.427217 14946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.432411 14946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37119
I20260927 02:55:04.432468 15074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37119 every 8 connection(s)
I20260927 02:55:04.432758 14946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:04.439524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 14946
I20260927 02:55:04.440915 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.433322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55777 (local address 127.0.98.62:41093)
0927 02:55:04.433405 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.433408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.433575 (+ 167us) server_negotiation.cc:407] Connection header received
0927 02:55:04.434008 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.434009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.434019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.434064 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:04.434955 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.435423 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.436313 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.436492 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.436537 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.438506 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:04.438508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:04.438508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:04.438515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:04.439667 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.440072 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.440073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.440074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.440124 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.440327 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.440328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.440328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.440432 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:04.440485 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.440735 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.440822 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":53}
I20260927 02:55:04.441059 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.433312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:55777)
0927 02:55:04.433459 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.433492 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.433935 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.434120 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.434123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.434644 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:04.434891 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.434897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.435482 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.435482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.436252 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.436254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.436547 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.436548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.436703 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.437100 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:04.437107 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:04.438413 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:04.439734 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:04.439740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:04.439742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:04.440001 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:04.440175 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:04.440176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:04.440177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:04.440274 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:04.440484 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:04.440490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:04.440617 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.440715 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.440856 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260927 02:55:04.441521 15075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41093
I20260927 02:55:04.441596 15075 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:04.441731 15075 heartbeater.cc:511] Master 127.0.98.62:41093 requested a full tablet report, sending...
I20260927 02:55:04.442065 14617 ts_manager.cc:194] Registered new tserver with Master: 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119)
I20260927 02:55:04.442462 14617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55777
I20260927 02:55:04.445796 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.439873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.0.98.62:41093)
0927 02:55:04.439978 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.439980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.440035 (+ 55us) server_negotiation.cc:407] Connection header received
0927 02:55:04.440118 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.440119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.440130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.440185 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:04.440731 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.441449 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.442604 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.442865 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.442922 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.443958 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:04.443959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:04.443960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:04.443965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:04.444983 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.445278 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.445278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.445279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.445320 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:04.445456 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:04.445456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:04.445456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:04.445536 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:04.445649 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.445681 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.445736 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
I20260927 02:55:04.446368 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:55:04.448777 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.446634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.0.98.62:41093)
0927 02:55:04.446689 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.446691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.446791 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:55:04.446880 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.446880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.446904 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.446954 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:55:04.447055 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.447476 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.448204 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.448375 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.448419 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.448584 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:55:04.448661 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:55:04.448710 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.448722 (+ 12us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.448735 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:55:04.449013 14617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:55:04.450525 14617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:55:04.452379 14739 tablet_service.cc:1467] Processing CreateTablet for tablet afa3b8ac4d60482a867113b6149a93ad (DEFAULT_TABLE table=test-table [id=8179a912e88b40d2b7da7aebb5f79bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:04.452570 14739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa3b8ac4d60482a867113b6149a93ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:04.454079 14697 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Bootstrap starting.
I20260927 02:55:04.454331 14697 tablet_bootstrap.cc:654] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:04.455029 14697 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: No bootstrap required, opened a new log
I20260927 02:55:04.455070 14697 ts_tablet_manager.cc:1397] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:55:04.455214 14697 raft_consensus.cc:348] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.455281 14697 raft_consensus.cc:374] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:04.455320 14697 raft_consensus.cc:729] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f6eba88737498299c625823b425834, State: Initialized, Role: FOLLOWER
I20260927 02:55:04.455358 14697 consensus_queue.cc:260] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.455423 14810 raft_consensus.cc:482] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:04.455498 14810 raft_consensus.cc:504] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.455736 14697 ts_tablet_manager.cc:1428] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:55:04.455792 14810 leader_election.cc:288] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203), 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119)
I20260927 02:55:04.456101 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.452117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34203 (local address 127.0.0.1:55326)
0927 02:55:04.452157 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.452167 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.452197 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.452722 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.452723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.452725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:04.452762 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.452763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.453822 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.453823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.454976 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.454978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.455876 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.455877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.455949 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.455971 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.455997 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.456037 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:55:04.456243 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.452361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.0.98.3:37119)
0927 02:55:04.452617 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.452621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.452631 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:55:04.452663 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.452665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.452678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.452754 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:04.452927 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.453919 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.455152 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.455804 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.455849 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.455904 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.456111 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.456165 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":72,"threads_started":1}
I20260927 02:55:04.456408 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.452374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.0.98.2:34203)
0927 02:55:04.452545 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.452550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.452560 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:55:04.452596 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.452598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.452613 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.452717 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:04.452815 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.453745 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.455057 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.455779 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.456195 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.456267 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.456292 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.456331 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260927 02:55:04.456437 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.452157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37119 (local address 127.0.0.1:41692)
0927 02:55:04.452286 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.452294 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.452326 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.452820 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.452821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.452823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:04.452862 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.452864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.454016 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.454016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.455066 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.455068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.455960 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.455961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.456023 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.456027 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.456331 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.456375 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260927 02:55:04.457017 14877 tablet_service.cc:1467] Processing CreateTablet for tablet afa3b8ac4d60482a867113b6149a93ad (DEFAULT_TABLE table=test-table [id=8179a912e88b40d2b7da7aebb5f79bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:04.457188 15009 tablet_service.cc:1467] Processing CreateTablet for tablet afa3b8ac4d60482a867113b6149a93ad (DEFAULT_TABLE table=test-table [id=8179a912e88b40d2b7da7aebb5f79bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:04.457334 14877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa3b8ac4d60482a867113b6149a93ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:04.457501 15009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa3b8ac4d60482a867113b6149a93ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:04.459265 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.455953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37119 (local address 127.0.98.1:48721)
0927 02:55:04.456036 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.456045 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.456083 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.456305 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.456306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.456308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:04.456361 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.456362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.457202 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.457203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.458257 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.458260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.458997 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.458998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.459085 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.459104 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.459159 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.459202 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 02:55:04.459663 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.456054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48721 (local address 127.0.98.3:37119)
0927 02:55:04.456178 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.456180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.456187 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:55:04.456205 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.456206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.456215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.456416 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:04.456444 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.457107 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.458351 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.458892 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.458980 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.458991 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.459531 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.459571 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
W20260927 02:55:04.460270 14692 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
I20260927 02:55:04.460732 15090 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Bootstrap starting.
I20260927 02:55:04.460872 15089 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: Bootstrap starting.
I20260927 02:55:04.461421 15090 tablet_bootstrap.cc:654] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:04.461745 15089 tablet_bootstrap.cc:654] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:04.461923 15090 log.cc:824] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:04.462059 15089 log.cc:824] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:04.462373 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.456216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34203 (local address 127.0.98.1:33997)
0927 02:55:04.459046 (+ 2830us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.459057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.459115 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.459276 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.459277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.459279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:04.459342 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.459344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.460174 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.460175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.461218 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.461221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.462045 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.462047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.462176 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.462181 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.462243 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.462288 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2813,"thread_start_us":63,"threads_started":1}
I20260927 02:55:04.462395 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.456495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33997 (local address 127.0.98.2:34203)
0927 02:55:04.456632 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.456634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.459163 (+ 2529us) server_negotiation.cc:407] Connection header received
0927 02:55:04.459183 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.459184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.459192 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.459243 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:04.459411 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.460081 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.461309 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.461915 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.462000 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.462010 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.462270 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.462309 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260927 02:55:04.462695 15090 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: No bootstrap required, opened a new log
I20260927 02:55:04.462749 15090 ts_tablet_manager.cc:1397] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:55:04.462944 15089 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: No bootstrap required, opened a new log
I20260927 02:55:04.463017 15089 ts_tablet_manager.cc:1397] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:55:04.463148 14897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3b8ac4d60482a867113b6149a93ad" candidate_uuid: "92f6eba88737498299c625823b425834" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9d583b24b74ceea0564a13a95713f0" is_pre_election: true
W20260927 02:55:04.463948 14691 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203): Illegal state: must be running to vote when last-logged opid is not known
I20260927 02:55:04.464006 14691 leader_election.cc:302] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f6eba88737498299c625823b425834; no voters: 6d117946453444bd80b8039c60e49cf7, af9d583b24b74ceea0564a13a95713f0
I20260927 02:55:04.464140 15085 raft_consensus.cc:2726] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 02:55:04.464390 15090 raft_consensus.cc:348] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.464497 15090 raft_consensus.cc:374] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:04.464519 15090 raft_consensus.cc:729] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9d583b24b74ceea0564a13a95713f0, State: Initialized, Role: FOLLOWER
I20260927 02:55:04.464601 15090 consensus_queue.cc:260] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.464830 15090 ts_tablet_manager.cc:1428] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:55:04.464879 14943 heartbeater.cc:503] Master 127.0.98.62:41093 was elected leader, sending a full tablet report...
I20260927 02:55:04.464781 15089 raft_consensus.cc:348] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.464943 15089 raft_consensus.cc:374] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:04.464979 15089 raft_consensus.cc:729] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d117946453444bd80b8039c60e49cf7, State: Initialized, Role: FOLLOWER
I20260927 02:55:04.465094 15089 consensus_queue.cc:260] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.465346 15089 ts_tablet_manager.cc:1428] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:55:04.465523 15075 heartbeater.cc:503] Master 127.0.98.62:41093 was elected leader, sending a full tablet report...
W20260927 02:55:04.557622 14944 tablet.cc:2369] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:55:04.591643 15094 raft_consensus.cc:482] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:04.591750 15094 raft_consensus.cc:504] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.592049 15094 leader_election.cc:288] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203), 92f6eba88737498299c625823b425834 (127.0.98.1:41303)
I20260927 02:55:04.594926 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.592294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47709 (local address 127.0.98.2:34203)
0927 02:55:04.592342 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.592344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.592496 (+ 152us) server_negotiation.cc:407] Connection header received
0927 02:55:04.592547 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.592549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.592561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.592606 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:04.592773 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.593224 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.594218 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.594581 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.594629 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.594640 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.594816 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.594856 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:55:04.594934 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.592244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34203 (local address 127.0.98.3:47709)
0927 02:55:04.592425 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.592435 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.592496 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.592619 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.592619 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.592621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:04.592706 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.592708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.593298 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.593299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.594126 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.594128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.594654 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.594654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.594719 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.594722 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.594807 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.594846 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":46,"thread_start_us":119,"threads_started":1}
I20260927 02:55:04.595011 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.592332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57447 (local address 127.0.98.1:41303)
0927 02:55:04.592431 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.592433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.592443 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:55:04.592490 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.592491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.592499 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.592553 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:04.592716 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.593229 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.594233 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.594655 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.594718 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.594815 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.594908 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.594953 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 02:55:04.595223 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.592241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.98.3:57447)
0927 02:55:04.592303 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.592314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.592376 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:04.592563 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:04.592564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:04.592566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:04.592633 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.592648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.593301 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.593302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.594140 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:04.594142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.594724 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.594725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:04.594843 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.594846 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:55:04.594898 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:55:04.594932 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":41}
I20260927 02:55:04.595280 14897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3b8ac4d60482a867113b6149a93ad" candidate_uuid: "6d117946453444bd80b8039c60e49cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9d583b24b74ceea0564a13a95713f0" is_pre_election: true
I20260927 02:55:04.595441 14897 raft_consensus.cc:2454] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d117946453444bd80b8039c60e49cf7 in term 0.
I20260927 02:55:04.595461 14758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3b8ac4d60482a867113b6149a93ad" candidate_uuid: "6d117946453444bd80b8039c60e49cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f6eba88737498299c625823b425834" is_pre_election: true
I20260927 02:55:04.595570 14758 raft_consensus.cc:2454] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d117946453444bd80b8039c60e49cf7 in term 0.
I20260927 02:55:04.595628 14962 leader_election.cc:302] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d117946453444bd80b8039c60e49cf7, af9d583b24b74ceea0564a13a95713f0; no voters:
I20260927 02:55:04.595744 15094 raft_consensus.cc:2781] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:55:04.595799 15094 raft_consensus.cc:482] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:55:04.595815 15094 raft_consensus.cc:3037] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:04.596535 15094 raft_consensus.cc:504] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.596650 15094 leader_election.cc:288] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [CANDIDATE]: Term 1 election: Requested vote from peers af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203), 92f6eba88737498299c625823b425834 (127.0.98.1:41303)
I20260927 02:55:04.596769 14897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3b8ac4d60482a867113b6149a93ad" candidate_uuid: "6d117946453444bd80b8039c60e49cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9d583b24b74ceea0564a13a95713f0"
I20260927 02:55:04.596788 14758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3b8ac4d60482a867113b6149a93ad" candidate_uuid: "6d117946453444bd80b8039c60e49cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f6eba88737498299c625823b425834"
I20260927 02:55:04.596843 14897 raft_consensus.cc:3037] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:04.596863 14758 raft_consensus.cc:3037] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:04.597481 14897 raft_consensus.cc:2454] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d117946453444bd80b8039c60e49cf7 in term 1.
I20260927 02:55:04.597645 14758 raft_consensus.cc:2454] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d117946453444bd80b8039c60e49cf7 in term 1.
I20260927 02:55:04.597643 14962 leader_election.cc:302] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d117946453444bd80b8039c60e49cf7, af9d583b24b74ceea0564a13a95713f0; no voters:
I20260927 02:55:04.597723 15094 raft_consensus.cc:2781] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:55:04.597851 15094 raft_consensus.cc:686] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 1 LEADER]: Becoming Leader. State: Replica: 6d117946453444bd80b8039c60e49cf7, State: Running, Role: LEADER
I20260927 02:55:04.597935 15094 consensus_queue.cc:237] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.599689 14616 catalog_manager.cc:5495] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d117946453444bd80b8039c60e49cf7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6d117946453444bd80b8039c60e49cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } health_report { overall_health: UNKNOWN } } }
I20260927 02:55:04.639180 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.637129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.0.98.3:37119)
0927 02:55:04.637193 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:04.637195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:04.637204 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:55:04.637249 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:04.637250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:04.637269 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:04.637316 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:55:04.637409 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.637884 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.638521 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:04.638698 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:04.638743 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:04.638915 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:55:04.639014 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:55:04.639085 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:04.639101 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 02:55:04.639122 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:55:04.640834 14897 raft_consensus.cc:1261] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 1 FOLLOWER]: Refusing update from remote peer 6d117946453444bd80b8039c60e49cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:55:04.640852 14758 raft_consensus.cc:1261] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Refusing update from remote peer 6d117946453444bd80b8039c60e49cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:55:04.641142 15094 consensus_queue.cc:1035] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:55:04.641275 15096 consensus_queue.cc:1035] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:55:04.643110 15098 mvcc.cc:205] Tried to move back new op lower bound from 7333796678206431232 to 7333796678033862656. Current Snapshot: MvccSnapshot[applied={T|T < 7333796678206431232}]
I20260927 02:55:04.645546 14616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:55:04.645624 14616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:55:04.646260 14616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Sending DeleteTablet for 3 replicas of tablet afa3b8ac4d60482a867113b6149a93ad
I20260927 02:55:04.646523 15009 tablet_service.cc:1514] Processing DeleteTablet for tablet afa3b8ac4d60482a867113b6149a93ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41692
I20260927 02:55:04.646538 14739 tablet_service.cc:1514] Processing DeleteTablet for tablet afa3b8ac4d60482a867113b6149a93ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47010
I20260927 02:55:04.646541 14877 tablet_service.cc:1514] Processing DeleteTablet for tablet afa3b8ac4d60482a867113b6149a93ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55326
I20260927 02:55:04.646703 14814 tablet_replica.cc:330] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: stopping tablet replica
I20260927 02:55:04.646740 15109 tablet_replica.cc:330] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: stopping tablet replica
I20260927 02:55:04.646778 14814 raft_consensus.cc:2229] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:55:04.646821 14814 raft_consensus.cc:2258] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:04.646847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 14655
I20260927 02:55:04.646845 15109 raft_consensus.cc:2229] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:55:04.646930 15109 raft_consensus.cc:2258] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:04.646966 15110 tablet_replica.cc:330] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: stopping tablet replica
I20260927 02:55:04.647078 15110 raft_consensus.cc:2229] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:55:04.647455 15109 ts_tablet_manager.cc:1905] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:04.648993 15110 raft_consensus.cc:2258] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:04.649101 15109 ts_tablet_manager.cc:1918] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:04.649176 15109 log.cc:1196] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/afa3b8ac4d60482a867113b6149a93ad
I20260927 02:55:04.649433 15110 ts_tablet_manager.cc:1905] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:04.649454 15109 ts_tablet_manager.cc:1939] T afa3b8ac4d60482a867113b6149a93ad P 6d117946453444bd80b8039c60e49cf7: Deleting consensus metadata
I20260927 02:55:04.649814 14602 catalog_manager.cc:4841] TS 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119): tablet afa3b8ac4d60482a867113b6149a93ad (table test-table [id=8179a912e88b40d2b7da7aebb5f79bf4]) successfully deleted
I20260927 02:55:04.651014 15110 ts_tablet_manager.cc:1918] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:04.651082 15110 log.cc:1196] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/afa3b8ac4d60482a867113b6149a93ad
I20260927 02:55:04.651345 15110 ts_tablet_manager.cc:1939] T afa3b8ac4d60482a867113b6149a93ad P af9d583b24b74ceea0564a13a95713f0: Deleting consensus metadata
I20260927 02:55:04.651705 14601 catalog_manager.cc:4841] TS af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203): tablet afa3b8ac4d60482a867113b6149a93ad (table test-table [id=8179a912e88b40d2b7da7aebb5f79bf4]) successfully deleted
W20260927 02:55:04.652645 14603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41303 (error 108)
I20260927 02:55:04.653019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 14811
I20260927 02:55:04.657622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 14946
I20260927 02:55:04.657711 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:04.657595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41303 (local address 127.0.0.1:47014)
0927 02:55:04.657639 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.657681 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:55:04.658933 14603 catalog_manager.cc:4584] TS 92f6eba88737498299c625823b425834 (127.0.98.1:41303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afa3b8ac4d60482a867113b6149a93ad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41303: connect: Connection refused (error 111)
I20260927 02:55:04.661470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 14584
I20260927 02:55:04.665719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41093
--webserver_interface=127.0.98.62
--webserver_port=45269
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:04.742122 15112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:04.742308 15112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:04.742337 15112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:04.743451 15112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:55:04.743489 15112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:04.743506 15112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:04.743521 15112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:55:04.743536 15112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:55:04.744935 15112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:04.745157 15112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:04.745355 15112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:04.749641 15117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.749756 15118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.749661 15120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:04.749992 15112 server_base.cc:1034] running on GCE node
I20260927 02:55:04.750200 15112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:04.750419 15112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:04.751559 15112 hybrid_clock.cc:648] HybridClock initialized: now 1790477704751546 us; error 29 us; skew 500 ppm
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:04.753762 15112 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:55:04.754140 15112 webserver.cc:460] Webserver started at http://127.0.98.62:45269/ using document root <none> and password file <none>
I20260927 02:55:04.754338 15112 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:04.754380 15112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:04.755590 15112 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:04.756129 15127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.756249 15112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:04.756319 15112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "401f8d880bb34f8f80683113d85950f6"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "9afe4a3979e8698f9c174a2d86239fb5"
server_key_iv: "5567f1db9aac40bd21b6fd709c810d2f"
server_key_version: "encryptionkey@0"
I20260927 02:55:04.756631 15112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:04.780803 15112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:04.782842 15112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:04.782979 15112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:04.786408 15112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41093
I20260927 02:55:04.786455 15179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41093 every 8 connection(s)
I20260927 02:55:04.786787 15112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:55:04.788198 15180 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:55:04.788800 15180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Bootstrap starting.
I20260927 02:55:04.789765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 15112
I20260927 02:55:04.790046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41303
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41093
--webserver_port=40367
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:55:04.796882 15180 log.cc:824] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:04.798892 15180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:55:04.799122 15180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: Bootstrap complete.
I20260927 02:55:04.801076 15180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:04.801287 15180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401f8d880bb34f8f80683113d85950f6, State: Initialized, Role: FOLLOWER
I20260927 02:55:04.801424 15180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:04.801493 15180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:55:04.801539 15180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:55:04.801582 15180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:55:04.802460 15180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:04.802547 15180 leader_election.cc:302] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401f8d880bb34f8f80683113d85950f6; no voters:
I20260927 02:55:04.802690 15180 leader_election.cc:288] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:55:04.802732 15184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:55:04.802861 15184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [term 3 LEADER]: Becoming Leader. State: Replica: 401f8d880bb34f8f80683113d85950f6, State: Running, Role: LEADER
I20260927 02:55:04.802942 15184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 02:55:04.802947 15180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:55:04.804126 15185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "401f8d880bb34f8f80683113d85950f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } } }
I20260927 02:55:04.804188 15186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401f8d880bb34f8f80683113d85950f6. Latest consensus state: current_term: 3 leader_uuid: "401f8d880bb34f8f80683113d85950f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401f8d880bb34f8f80683113d85950f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } } }
I20260927 02:55:04.804349 15185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:04.804352 15186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:04.804754 15199 catalog_manager.cc:1259] Loaded cluster ID: e31bb802857244fd88a4ac71818bf8ec
I20260927 02:55:04.804824 15199 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: loading cluster ID for follower catalog manager: success
W20260927 02:55:04.805073 15199 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 401f8d880bb34f8f80683113d85950f6: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260927 02:55:04.805115 15199 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260927 02:55:04.806875 15200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:55:04.807307 15200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:55:04.807392 15200 catalog_manager.cc:1259] Loaded cluster ID: e31bb802857244fd88a4ac71818bf8ec
I20260927 02:55:04.807423 15200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 02:55:04.807590 15200 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f65713f29ad google::LogMessage::Fail() at ??:0
@ 0x7f65713f588a google::LogMessage::SendToLog() at ??:0
@ 0x7f65713f24de google::LogMessage::Flush() at ??:0
@ 0x7f65713f3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f65793170f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f65793208b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f65732c00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f65732b5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f65741196db start_thread at ??:0
@ 0x7f656fef061f clone at ??:0
W20260927 02:55:04.878544 15182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:04.878726 15182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:04.878746 15182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:04.879930 15182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:04.879973 15182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:04.879997 15182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:55:04.881402 15182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40367
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:04.881603 15182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:04.881798 15182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:04.886320 15216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.886345 15217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:04.886560 15219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:04.886580 15182 server_base.cc:1034] running on GCE node
I20260927 02:55:04.886831 15182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:04.887056 15182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:04.888198 15182 hybrid_clock.cc:648] HybridClock initialized: now 1790477704888190 us; error 27 us; skew 500 ppm
Sep 27 02:55:04 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:04.890442 15182 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:55:04.890911 15182 webserver.cc:460] Webserver started at http://127.0.98.1:40367/ using document root <none> and password file <none>
I20260927 02:55:04.891114 15182 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:04.891155 15182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:04.892383 15182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:04.892982 15226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:04.893143 15182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:04.893209 15182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "92f6eba88737498299c625823b425834"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "c0e9a40520f56c905b65b2c5af8fa717"
server_key_iv: "6ab70b0d619250975a929f3aa908078e"
server_key_version: "encryptionkey@0"
I20260927 02:55:04.893529 15182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:04.924367 15182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:04.926492 15182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:04.926599 15182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:04.926854 15182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:04.927261 15233 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:55:04.929090 15182 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:55:04.929133 15182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:55:04.929163 15182 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:55:04.929756 15182 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:55:04.929785 15182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:55:04.929877 15233 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Bootstrap starting.
I20260927 02:55:04.936359 15182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41303
I20260927 02:55:04.936430 15340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41303 every 8 connection(s)
I20260927 02:55:04.936723 15182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:55:04.940622 15233 log.cc:824] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:04.942654 15233 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:55:04.942904 15233 tablet_bootstrap.cc:492] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Bootstrap complete.
I20260927 02:55:04.943621 15233 ts_tablet_manager.cc:1397] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 02:55:04.944537 15233 raft_consensus.cc:348] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.944747 15233 raft_consensus.cc:729] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f6eba88737498299c625823b425834, State: Initialized, Role: FOLLOWER
I20260927 02:55:04.944866 15233 consensus_queue.cc:260] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:04.945117 15233 ts_tablet_manager.cc:1428] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 02:55:04.945176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 15182
I20260927 02:55:04.945405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34203
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41093
--webserver_port=45647
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:05.044886 15343 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:55:04.937384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:47071)
0927 02:55:04.937551 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:04.937583 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:04.938001 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:05.044685 (+106684us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
W20260927 02:55:05.045120 15341 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41093 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41093: Client connection negotiation failed: client connection to 127.0.98.62:41093: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:55:05.045442 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:05.045339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:41073)
0927 02:55:05.045385 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:05.045421 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:55:05.045778 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:05.045702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:47953)
0927 02:55:05.045743 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:05.045764 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:55:05.045876 15341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:55:05.053911 15346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:05.054101 15346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:05.054133 15346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:05.055286 15346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:05.055322 15346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:05.055357 15346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:55:05.056773 15346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45647
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:05.056995 15346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:05.057194 15346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:05.061866 15352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:05.061968 15346 server_base.cc:1034] running on GCE node
W20260927 02:55:05.061928 15351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:05.061877 15354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:05.062322 15346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:05.062518 15346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:05.063642 15346 hybrid_clock.cc:648] HybridClock initialized: now 1790477705063626 us; error 32 us; skew 500 ppm
Sep 27 02:55:05 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:05.065928 15346 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:55:05.066288 15346 webserver.cc:460] Webserver started at http://127.0.98.2:45647/ using document root <none> and password file <none>
I20260927 02:55:05.066459 15346 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:05.066496 15346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:05.067718 15346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:05.068181 15361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:05.068311 15346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:05.068370 15346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "af9d583b24b74ceea0564a13a95713f0"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "cf8fd670b52737d07499a11093362915"
server_key_iv: "7ba1587d85eca454cbd5c8d741435505"
server_key_version: "encryptionkey@0"
I20260927 02:55:05.068691 15346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:05.080932 15346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:05.082919 15346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:05.083026 15346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:05.083228 15346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:05.083484 15346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:05.083513 15346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:05.083544 15346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:05.083566 15346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:05.088963 15346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34203
I20260927 02:55:05.089066 15474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34203 every 8 connection(s)
I20260927 02:55:05.089326 15346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:55:05.090281 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:05.089944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:42075)
0927 02:55:05.090076 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:05.090143 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260927 02:55:05.090425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 15346
I20260927 02:55:05.090641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37119
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41093
--webserver_port=39705
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:05.090680 15475 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41093 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41093: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
I20260927 02:55:05.090989 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:05.090908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:57719)
0927 02:55:05.090948 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:05.090973 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:55:05.091216 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:05.091147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:34313)
0927 02:55:05.091181 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:05.091202 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:55:05.091307 15475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:55:05.171105 15478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:05.171291 15478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:05.171312 15478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:05.172483 15478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:05.172537 15478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:05.172574 15478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:55:05.173980 15478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39705
--tserver_master_addrs=127.0.98.62:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:05.174160 15478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:05.174329 15478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:05.179131 15484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:05.179102 15486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:05.179090 15483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:05.179342 15478 server_base.cc:1034] running on GCE node
I20260927 02:55:05.179534 15478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:05.179752 15478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:05.180909 15478 hybrid_clock.cc:648] HybridClock initialized: now 1790477705180896 us; error 31 us; skew 500 ppm
Sep 27 02:55:05 dist-test-slave-mfx9 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:05.183265 15478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:55:05.183634 15478 webserver.cc:460] Webserver started at http://127.0.98.3:39705/ using document root <none> and password file <none>
I20260927 02:55:05.183794 15478 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:05.183831 15478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:05.185050 15478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:05.185504 15493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:05.185667 15478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:05.185745 15478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "6d117946453444bd80b8039c60e49cf7"
format_stamp: "Formatted at 2026-09-27 02:55:03 on dist-test-slave-mfx9"
server_key: "146435a53bc103484221f4a2b6cd9ac3"
server_key_iv: "9c87d8c60fd6574902e2210eea1a2354"
server_key_version: "encryptionkey@0"
I20260927 02:55:05.186038 15478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:05.200212 15478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:05.201787 15478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:05.201903 15478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:05.202086 15478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:05.202348 15478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:05.202378 15478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:05.202399 15478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:05.202420 15478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:05.207625 15478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37119
I20260927 02:55:05.207687 15606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37119 every 8 connection(s)
I20260927 02:55:05.207969 15478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:55:05.208863 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:05.208549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:42029)
0927 02:55:05.208673 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:05.208734 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
W20260927 02:55:05.209183 15607 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41093 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41093: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
I20260927 02:55:05.209412 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:05.209336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:57869)
0927 02:55:05.209372 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:05.209396 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 02:55:05.209628 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:05.209566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:44547)
0927 02:55:05.209594 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:05.209617 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 02:55:05.209702 15607 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:55:05.215058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 15478
I20260927 02:55:06.046365 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:06.046159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:47543)
0927 02:55:06.046272 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:06.046304 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260927 02:55:06.091769 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:06.091594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:39869)
0927 02:55:06.091688 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:06.091719 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
I20260927 02:55:06.210161 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:06.209970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:37697)
0927 02:55:06.210073 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:06.210115 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260927 02:55:06.427407 15614 raft_consensus.cc:482] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:06.427515 15614 raft_consensus.cc:504] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:06.427840 15614 leader_election.cc:288] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203), 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119)
Sep 27 02:55:06 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 02:55:06 dist-test-slave-mfx9 krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 02:55:06.437170 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:06.428113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38099 (local address 127.0.98.3:37119)
0927 02:55:06.428275 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:06.428283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:06.428308 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:55:06.428727 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:06.428729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:06.428766 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:06.428883 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:06.429592 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:06.430415 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:06.431411 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:06.431797 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:06.431890 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:06.434334 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:06.434338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:06.434338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:06.434349 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:06.435789 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:06.436113 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:06.436116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:06.436116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:06.436190 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:06.436413 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:06.436414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:06.436414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:06.436621 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:06.436690 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:06.436952 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 02:55:06.437062 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260927 02:55:06.437175 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:06.428088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41301 (local address 127.0.98.2:34203)
0927 02:55:06.428258 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:06.428268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:06.428294 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:55:06.428723 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:06.428726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:06.428766 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:06.428883 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:06.429596 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:06.430418 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:06.431412 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:06.431797 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:06.431890 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:06.433984 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:06.433988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:06.433988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:06.433999 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:06.435668 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:06.436084 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:06.436087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:06.436087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:06.436188 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:06.436417 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:06.436418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:06.436418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:06.436636 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:06.436705 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:06.436953 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:55:06.437073 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260927 02:55:06.437170 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:06.427966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34203 (local address 127.0.98.1:41301)
0927 02:55:06.428033 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:06.428044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:06.428098 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:06.428936 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:06.428938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:06.429305 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:06.429490 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:06.429496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:06.430517 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:06.430517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:06.431335 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:06.431338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:06.431875 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:06.431876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:06.431978 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:06.432482 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:06.432492 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:06.433865 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:06.435745 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:06.435751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:06.435753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:06.436018 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:06.436248 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:06.436248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:06.436249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:06.436360 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:06.436700 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:06.436701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:06.436810 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:55:06.436893 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:55:06.437063 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":55}
I20260927 02:55:06.437248 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:06.427966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37119 (local address 127.0.98.1:38099)
0927 02:55:06.428095 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:06.428105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:06.428167 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:06.428935 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:06.428938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:06.429304 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:06.429490 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:06.429496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:06.430503 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:06.430504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:06.431336 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:06.431338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:06.431877 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:06.431878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:06.431979 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:06.432516 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:06.432519 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:06.434248 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:06.435863 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:06.435864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:06.435864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:06.436045 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:06.436247 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:06.436248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:06.436248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:06.436356 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:06.436690 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:06.436696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:06.436813 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 02:55:06.436893 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:55:06.437139 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":170,"thread_start_us":62,"threads_started":1}
W20260927 02:55:06.437841 15227 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
W20260927 02:55:06.438520 15228 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
I20260927 02:55:06.438567 15228 leader_election.cc:302] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f6eba88737498299c625823b425834; no voters: 6d117946453444bd80b8039c60e49cf7, af9d583b24b74ceea0564a13a95713f0
I20260927 02:55:06.438666 15614 raft_consensus.cc:2726] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:55:07.046993 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:07.046792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:33049)
0927 02:55:07.046913 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:07.046946 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260927 02:55:07.092363 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:07.092180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:47413)
0927 02:55:07.092288 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:07.092324 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260927 02:55:07.210664 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:07.210490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:51425)
0927 02:55:07.210594 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:07.210624 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260927 02:55:08.047649 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:08.047404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:47571)
0927 02:55:08.047552 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:08.047598 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260927 02:55:08.092911 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:08.092739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:46007)
0927 02:55:08.092845 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:08.092875 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260927 02:55:08.211226 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:08.211037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:58577)
0927 02:55:08.211145 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:08.211181 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260927 02:55:08.304823 15624 raft_consensus.cc:482] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:08.304905 15624 raft_consensus.cc:504] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:08.305037 15624 leader_election.cc:288] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203), 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119)
W20260927 02:55:08.305382 15228 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
W20260927 02:55:08.305503 15227 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
I20260927 02:55:08.305557 15227 leader_election.cc:302] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f6eba88737498299c625823b425834; no voters: 6d117946453444bd80b8039c60e49cf7, af9d583b24b74ceea0564a13a95713f0
I20260927 02:55:08.305619 15624 raft_consensus.cc:2726] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:55:09.048190 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:09.048013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:48473)
0927 02:55:09.048121 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:09.048149 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260927 02:55:09.093472 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:09.093300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:35535)
0927 02:55:09.093400 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:09.093433 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260927 02:55:09.211706 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:09.211543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:60559)
0927 02:55:09.211641 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:09.211669 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260927 02:55:10.048911 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:10.048693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:55391)
0927 02:55:10.048825 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:10.048859 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260927 02:55:10.094012 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:10.093815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:56755)
0927 02:55:10.093938 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:10.093972 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260927 02:55:10.212255 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:10.212052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:52883)
0927 02:55:10.212187 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:10.212219 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260927 02:55:11.011590 15631 raft_consensus.cc:482] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:11.011672 15631 raft_consensus.cc:504] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:11.011823 15631 leader_election.cc:288] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203), 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119)
W20260927 02:55:11.012146 15228 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
W20260927 02:55:11.012229 15227 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
I20260927 02:55:11.012274 15227 leader_election.cc:302] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f6eba88737498299c625823b425834; no voters: 6d117946453444bd80b8039c60e49cf7, af9d583b24b74ceea0564a13a95713f0
I20260927 02:55:11.012334 15631 raft_consensus.cc:2726] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:55:11.049520 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:11.049337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:50849)
0927 02:55:11.049443 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:11.049476 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260927 02:55:11.094599 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:11.094386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:47873)
0927 02:55:11.094500 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:11.094553 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260927 02:55:11.212761 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:11.212596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:37735)
0927 02:55:11.212696 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:11.212724 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260927 02:55:12.050117 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:12.049923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:59571)
0927 02:55:12.050038 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:12.050071 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260927 02:55:12.095075 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:12.094904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:34101)
0927 02:55:12.095007 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:12.095040 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260927 02:55:12.213318 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:12.213129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:38133)
0927 02:55:12.213240 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:12.213275 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260927 02:55:13.050832 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:13.050553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:37075)
0927 02:55:13.050696 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:13.050738 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260927 02:55:13.095602 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:13.095423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:40027)
0927 02:55:13.095526 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:13.095558 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260927 02:55:13.213907 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:13.213726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:56447)
0927 02:55:13.213833 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:13.213866 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260927 02:55:13.765048 15641 raft_consensus.cc:482] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:13.765141 15641 raft_consensus.cc:504] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9d583b24b74ceea0564a13a95713f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34203 } } peers { permanent_uuid: "6d117946453444bd80b8039c60e49cf7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37119 } } peers { permanent_uuid: "92f6eba88737498299c625823b425834" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41303 } }
I20260927 02:55:13.765300 15641 leader_election.cc:288] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203), 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119)
W20260927 02:55:13.765606 15227 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af9d583b24b74ceea0564a13a95713f0 (127.0.98.2:34203): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
W20260927 02:55:13.765707 15228 leader_election.cc:341] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d117946453444bd80b8039c60e49cf7 (127.0.98.3:37119): Not found: Tablet not found: afa3b8ac4d60482a867113b6149a93ad
I20260927 02:55:13.765749 15228 leader_election.cc:302] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f6eba88737498299c625823b425834; no voters: 6d117946453444bd80b8039c60e49cf7, af9d583b24b74ceea0564a13a95713f0
I20260927 02:55:13.765818 15641 raft_consensus.cc:2726] T afa3b8ac4d60482a867113b6149a93ad P 92f6eba88737498299c625823b425834 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:55:14.051575 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:14.051346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:42587)
0927 02:55:14.051480 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:14.051528 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260927 02:55:14.096221 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:14.096016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:47311)
0927 02:55:14.096140 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:14.096181 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260927 02:55:14.214545 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:14.214328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:46401)
0927 02:55:14.214446 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:14.214488 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260927 02:55:15.052217 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:15.052024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:52333)
0927 02:55:15.052142 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:15.052173 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260927 02:55:15.096858 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:15.096623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:58421)
0927 02:55:15.096780 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:15.096815 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260927 02:55:15.215197 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:15.215008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:55899)
0927 02:55:15.215123 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:15.215157 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260927 02:55:16.052829 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:16.052619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:55229)
0927 02:55:16.052749 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:16.052784 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260927 02:55:16.097451 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:16.097255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:45749)
0927 02:55:16.097366 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:16.097411 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260927 02:55:16.215744 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:16.215572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:46105)
0927 02:55:16.215671 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:16.215702 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260927 02:55:17.053506 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:17.053304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:58927)
0927 02:55:17.053421 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:17.053460 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260927 02:55:17.098074 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:17.097867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:47119)
0927 02:55:17.097994 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:17.098030 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260927 02:55:17.216301 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:17.216098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:46325)
0927 02:55:17.216226 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:17.216262 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260927 02:55:18.054131 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:18.053923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:49355)
0927 02:55:18.054047 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:18.054086 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260927 02:55:18.098694 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:18.098498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:60069)
0927 02:55:18.098611 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:18.098649 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260927 02:55:18.216826 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:18.216652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:43417)
0927 02:55:18.216755 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:18.216786 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260927 02:55:19.054811 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:19.054567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:45717)
0927 02:55:19.054693 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:19.054736 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260927 02:55:19.099303 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:19.099102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:48421)
0927 02:55:19.099218 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:19.099252 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260927 02:55:19.217417 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:19.217225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:47201)
0927 02:55:19.217327 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:19.217359 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260927 02:55:20.055485 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:20.055261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.1:54885)
0927 02:55:20.055404 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:20.055441 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260927 02:55:20.099869 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:20.099676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.2:55883)
0927 02:55:20.099797 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:20.099829 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260927 02:55:20.218089 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:20.217882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41093 (local address 127.0.98.3:51437)
0927 02:55:20.217999 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:20.218038 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260927 02:55:20.218083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 15182
I20260927 02:55:20.222178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 15346
I20260927 02:55:20.225916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 15478
2026-09-27T02:55:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18162 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:55:20 dist-test-slave-mfx9 krb5kdc[15665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:55:20 dist-test-slave-mfx9 krb5kdc[15665](info): set up 2 sockets
Sep 27 02:55:20 dist-test-slave-mfx9 krb5kdc[15665](info): commencing operation
krb5kdc: starting...
W20260927 02:55:21.166950 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:55:21.271518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:21.347543 15678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:21.347733 15678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:21.347754 15678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:21.348812 15678 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:55:21.348845 15678 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:55:21.348855 15678 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:55:21.348865 15678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:55:21.348876 15678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:21.348892 15678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:21.348903 15678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:55:21.348920 15678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:55:21.350281 15678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:21.350453 15678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:21.350632 15678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:55:21.355024 15678 server_base.cc:1034] running on GCE node
W20260927 02:55:21.355010 15684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:21.355001 15683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:21.355010 15686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:21.355440 15678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:21.355695 15678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:21.356841 15678 hybrid_clock.cc:648] HybridClock initialized: now 1790477721356831 us; error 26 us; skew 500 ppm
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:21.359120 15678 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:55:21.359503 15678 webserver.cc:460] Webserver started at http://127.0.98.62:42775/ using document root <none> and password file <none>
I20260927 02:55:21.359666 15678 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:21.359704 15678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:21.359789 15678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:21.360734 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "34e0318197de404aa68c8c0bc1cd1b7a"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "eda5e24939292a449ab4e0df7c10664e"
server_key_iv: "4922fd07ebe8dad9710c5a042a29fbc9"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.361023 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "34e0318197de404aa68c8c0bc1cd1b7a"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "eda5e24939292a449ab4e0df7c10664e"
server_key_iv: "4922fd07ebe8dad9710c5a042a29fbc9"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.362124 15678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:55:21.362788 15693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.362938 15678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:21.363019 15678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "34e0318197de404aa68c8c0bc1cd1b7a"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "eda5e24939292a449ab4e0df7c10664e"
server_key_iv: "4922fd07ebe8dad9710c5a042a29fbc9"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.363081 15678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:21.382031 15678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:21.384068 15678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:21.384197 15678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:21.387794 15678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45791
I20260927 02:55:21.387833 15745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45791 every 8 connection(s)
I20260927 02:55:21.388144 15678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:55:21.388914 15746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:21.391062 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Bootstrap starting.
I20260927 02:55:21.391599 15746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:21.391850 15746 log.cc:824] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:21.392433 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: No bootstrap required, opened a new log
I20260927 02:55:21.393915 15746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:55:21.394048 15746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:21.394069 15746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e0318197de404aa68c8c0bc1cd1b7a, State: Initialized, Role: FOLLOWER
I20260927 02:55:21.394222 15746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:55:21.394282 15746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:55:21.394301 15746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:55:21.394344 15746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:21.394836 15746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:55:21.394919 15746 leader_election.cc:302] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34e0318197de404aa68c8c0bc1cd1b7a; no voters:
I20260927 02:55:21.395058 15746 leader_election.cc:288] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:55:21.395102 15749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:55:21.395242 15749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 1 LEADER]: Becoming Leader. State: Replica: 34e0318197de404aa68c8c0bc1cd1b7a, State: Running, Role: LEADER
I20260927 02:55:21.395313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 15678
I20260927 02:55:21.395294 15746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:55:21.395401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:55:21.395412 15749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:55:21.395597 392 external_mini_cluster.cc:1428] Setting key c78fc8631303006eb09ecaf5563a4c64
I20260927 02:55:21.396136 15750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260927 02:55:21.396164 15751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34e0318197de404aa68c8c0bc1cd1b7a. Latest consensus state: current_term: 1 leader_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260927 02:55:21.396215 15750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:21.396240 15751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:21.396708 15759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:55:21.397200 15759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:55:21.398702 15759 catalog_manager.cc:1347] Generated new cluster ID: 5564eb47483a4de2b0202bf057665d15
I20260927 02:55:21.398785 15759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:21.405014 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.396426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.0.98.62:45791)
0927 02:55:21.396666 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:21.396676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:21.396710 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:55:21.396822 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:21.396829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:21.396901 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:21.397040 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:21.397788 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.398837 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.399568 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.399987 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.400080 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.402160 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:21.402164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:21.402164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:21.402175 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:21.403664 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.403966 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.403969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.403970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.404059 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.404235 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.404235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.404236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.404454 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:21.404571 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:21.404697 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:55:21.404824 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260927 02:55:21.407711 15759 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:55:21.408352 15759 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:55:21.412369 15759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Generated new TSK 0
I20260927 02:55:21.412514 15759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:55:21.441855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:21.525046 15772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:21.525233 15772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:21.525262 15772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:21.526393 15772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:21.526435 15772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:21.526470 15772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:55:21.527925 15772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:21.528155 15772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:21.528357 15772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:21.532933 15777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:21.532945 15778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:21.533073 15772 server_base.cc:1034] running on GCE node
W20260927 02:55:21.532995 15780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:21.533313 15772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:21.533545 15772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:21.534687 15772 hybrid_clock.cc:648] HybridClock initialized: now 1790477721534679 us; error 31 us; skew 500 ppm
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:21.537036 15772 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:55:21.537422 15772 webserver.cc:460] Webserver started at http://127.0.98.1:40305/ using document root <none> and password file <none>
I20260927 02:55:21.537595 15772 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:21.537631 15772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:21.537717 15772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:21.538681 15772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd6001bd73ec4f61963644600e60619d"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "149fb10e898049a6135512fee543e908"
server_key_iv: "e58a3e0f2274a8c3754c17db185deac2"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.539008 15772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd6001bd73ec4f61963644600e60619d"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "149fb10e898049a6135512fee543e908"
server_key_iv: "e58a3e0f2274a8c3754c17db185deac2"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.540201 15772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:21.540855 15787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.540997 15772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:21.541075 15772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "dd6001bd73ec4f61963644600e60619d"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "149fb10e898049a6135512fee543e908"
server_key_iv: "e58a3e0f2274a8c3754c17db185deac2"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.541144 15772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:21.566815 15772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:21.568876 15772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:21.569001 15772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:21.569202 15772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:21.569481 15772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:21.569518 15772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.569540 15772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:21.569561 15772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.574781 15772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46257
I20260927 02:55:21.574878 15900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46257 every 8 connection(s)
I20260927 02:55:21.575126 15772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:55:21.575945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 15772
I20260927 02:55:21.576081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:55:21.576272 392 external_mini_cluster.cc:1428] Setting key 3eb59b24a3aa638c397f38d4cf69c322
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:55:21.584204 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.575781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40219 (local address 127.0.98.62:45791)
0927 02:55:21.575837 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:21.575839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:21.576073 (+ 234us) server_negotiation.cc:407] Connection header received
0927 02:55:21.576596 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:21.576597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:21.576610 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:21.576658 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:21.577584 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.578252 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.579300 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.579520 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.579576 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.581672 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:21.581673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:21.581673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:21.581680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:21.582946 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.583344 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.583345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.583346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.583400 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.583603 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.583604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.583604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.583708 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:21.583761 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:21.584039 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 02:55:21.584118 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:55:21.584368 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.575764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:40219)
0927 02:55:21.575931 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:21.575966 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:21.576503 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:21.576733 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:21.576737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:21.577299 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:21.577487 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.577492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.578341 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.578342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.579220 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.579223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.579587 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.579588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.579715 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.580162 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:21.580171 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:21.581585 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:21.583044 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:21.583050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:21.583053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:21.583281 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:21.583450 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:21.583451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:21.583451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:21.583550 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:21.583759 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:21.583765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:21.583924 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 02:55:21.584016 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:55:21.584165 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260927 02:55:21.584851 15901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260927 02:55:21.584939 15901 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:21.585098 15901 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:55:21.585639 15710 ts_manager.cc:194] Registered new tserver with Master: dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257)
I20260927 02:55:21.586218 15710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:55:21.597905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:21.674490 15906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:21.674676 15906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:21.674697 15906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:21.675813 15906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:21.675853 15906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:21.675877 15906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:55:21.677227 15906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:21.677419 15906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:21.677610 15906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:55:21.682188 15906 server_base.cc:1034] running on GCE node
W20260927 02:55:21.682200 15914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:21.682173 15912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:21.682165 15911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:21.682556 15906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:21.682741 15906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:21.683911 15906 hybrid_clock.cc:648] HybridClock initialized: now 1790477721683899 us; error 28 us; skew 500 ppm
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:21.686048 15906 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:55:21.686398 15906 webserver.cc:460] Webserver started at http://127.0.98.2:44497/ using document root <none> and password file <none>
I20260927 02:55:21.686551 15906 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:21.686589 15906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:21.686672 15906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:21.687611 15906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "0f948ebb84c144a38c3de9116cce83ed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "c487e51baa1c44d869291073e6ab8ae3"
server_key_iv: "3e2c6a99c300bde59e643ff76704ba04"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.687881 15906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0f948ebb84c144a38c3de9116cce83ed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "c487e51baa1c44d869291073e6ab8ae3"
server_key_iv: "3e2c6a99c300bde59e643ff76704ba04"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.688946 15906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:55:21.689543 15921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.689669 15906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.689744 15906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "0f948ebb84c144a38c3de9116cce83ed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "c487e51baa1c44d869291073e6ab8ae3"
server_key_iv: "3e2c6a99c300bde59e643ff76704ba04"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.689800 15906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:21.712050 15906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:21.713667 15906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:21.713791 15906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:21.713985 15906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:21.714267 15906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:21.714298 15906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.714325 15906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:21.714350 15906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.719581 15906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45319
I20260927 02:55:21.719640 16034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45319 every 8 connection(s)
I20260927 02:55:21.719903 15906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:55:21.721490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 15906
I20260927 02:55:21.721572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:55:21.721735 392 external_mini_cluster.cc:1428] Setting key eeadcf3180366ef243033a59cc81a0c9
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:21.728646 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.720485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33335 (local address 127.0.98.62:45791)
0927 02:55:21.720540 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:21.720542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:21.720734 (+ 192us) server_negotiation.cc:407] Connection header received
0927 02:55:21.721180 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:21.721181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:21.721190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:21.721242 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:21.722255 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.722956 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.723905 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.724190 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.724261 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.726174 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:21.726175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:21.726175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:21.726183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:21.727460 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.727808 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.727809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.727809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.727862 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.728069 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.728070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.728070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.728170 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:21.728224 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:21.728465 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 02:55:21.728560 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:55:21.728777 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.720457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:33335)
0927 02:55:21.720609 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:21.720659 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:21.721108 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:21.721294 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:21.721297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:21.721913 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:21.722170 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.722177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.723035 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.723035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.723827 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.723830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.724259 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.724259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.724349 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.724767 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:21.724774 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:21.725984 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:21.727538 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:21.727545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:21.727547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:21.727744 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:21.727914 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:21.727915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:21.727915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:21.728016 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:21.728222 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:21.728228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:21.728354 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:55:21.728437 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:55:21.728577 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260927 02:55:21.729176 16035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260927 02:55:21.729254 16035 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:55:21.729414 16035 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260927 02:55:21.729756 15710 ts_manager.cc:194] Registered new tserver with Master: 0f948ebb84c144a38c3de9116cce83ed (127.0.98.2:45319)
I20260927 02:55:21.730101 15710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:55:21.743275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:21.819228 16040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:21.819407 16040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:21.819428 16040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:21.820485 16040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:21.820515 16040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:21.820539 16040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:55:21.821875 16040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:21.822063 16040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:21.822268 16040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:21.826773 16046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:21.826787 16048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:21.826798 16045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:21.826817 16040 server_base.cc:1034] running on GCE node
I20260927 02:55:21.827175 16040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:21.827378 16040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:21.828512 16040 hybrid_clock.cc:648] HybridClock initialized: now 1790477721828497 us; error 28 us; skew 500 ppm
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:21.830689 16040 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:55:21.831077 16040 webserver.cc:460] Webserver started at http://127.0.98.3:38549/ using document root <none> and password file <none>
I20260927 02:55:21.831231 16040 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:21.831269 16040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:21.831350 16040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:55:21.832258 16040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "6af445c338d341fa85f6c4c8180b0aed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "670870ae6a24400b0c35f7c94a4e77db"
server_key_iv: "521cdb16902c6d51426337ce88d686a6"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.832523 16040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6af445c338d341fa85f6c4c8180b0aed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "670870ae6a24400b0c35f7c94a4e77db"
server_key_iv: "521cdb16902c6d51426337ce88d686a6"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.833681 16040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:55:21.834270 16055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.834419 16040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:21.834496 16040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "6af445c338d341fa85f6c4c8180b0aed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "670870ae6a24400b0c35f7c94a4e77db"
server_key_iv: "521cdb16902c6d51426337ce88d686a6"
server_key_version: "encryptionkey@0"
I20260927 02:55:21.834544 16040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:21.874620 16040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:21.876663 16040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:21.876768 16040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:21.876935 16040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:21.877224 16040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:21.877259 16040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.877279 16040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:21.877295 16040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:21.882469 16040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39173
I20260927 02:55:21.882545 16168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39173 every 8 connection(s)
I20260927 02:55:21.882808 16040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:55:21.887346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 16040
I20260927 02:55:21.887444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:55:21.887616 392 external_mini_cluster.cc:1428] Setting key 4d225a84400e6a21261fdde360645df1
Sep 27 02:55:21 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:21.891044 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.883453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57325 (local address 127.0.98.62:45791)
0927 02:55:21.883529 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:21.883531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:21.883689 (+ 158us) server_negotiation.cc:407] Connection header received
0927 02:55:21.884144 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:21.884145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:21.884155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:21.884188 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:21.885065 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.885542 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.886475 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.886656 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.886697 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.888721 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:21.888722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:21.888722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:21.888728 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:21.889992 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.890330 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.890330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.890331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.890375 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.890536 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.890536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.890537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.890636 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:21.890678 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:21.890923 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 02:55:21.890982 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:55:21.891235 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.883425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:57325)
0927 02:55:21.883582 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:21.883616 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:21.884060 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:21.884248 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:21.884251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:21.884775 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:21.884970 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.884975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.885625 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.885625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.886415 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.886417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.886704 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.886705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.886832 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.887215 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:21.887223 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:21.888640 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:21.890064 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:21.890070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:21.890072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:21.890272 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:21.890414 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:21.890414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:21.890415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:21.890498 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:21.890678 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:21.890681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:21.890816 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 02:55:21.890906 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:55:21.891044 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260927 02:55:21.891618 16169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260927 02:55:21.891700 16169 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:21.891855 16169 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260927 02:55:21.892182 15710 ts_manager.cc:194] Registered new tserver with Master: 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173)
I20260927 02:55:21.892551 15710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57325
I20260927 02:55:21.899140 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:55:21.905761 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.900367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.98.62:45791)
0927 02:55:21.900418 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:21.900421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:21.900505 (+ 84us) server_negotiation.cc:407] Connection header received
0927 02:55:21.900565 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:21.900566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:21.900582 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:21.900626 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:21.901159 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.901619 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.902317 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.902495 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.902546 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.903608 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:21.903609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:21.903610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:21.903616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:21.904759 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.905087 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.905088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.905088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.905138 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:21.905295 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:21.905296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:21.905296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:21.905423 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:21.905505 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:21.905608 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:55:21.905677 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:55:21.907076 15710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:55:21.908475 15710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:55:21.914067 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.910450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:58782)
0927 02:55:21.910620 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:21.910629 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:21.910678 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:21.910933 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:21.910934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:21.910936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:21.910986 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.910988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.911874 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.911875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.913072 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.913074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.913796 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.913797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.913873 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.913883 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 02:55:21.913938 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:55:21.913988 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":7,"thread_start_us":50,"threads_started":1}
I20260927 02:55:21.914078 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.910626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.0.98.2:45319)
0927 02:55:21.910805 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:21.910809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:21.910818 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:55:21.910853 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:21.910855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:21.910867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:21.910934 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:21.911038 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.911786 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.913161 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.913717 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.913775 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.913832 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:21.913959 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 02:55:21.914000 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260927 02:55:21.914286 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.910294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46257 (local address 127.0.0.1:51278)
0927 02:55:21.910586 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:21.910595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:21.910638 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:21.910843 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:21.910846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:21.910848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:21.910909 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.910911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.912215 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.912215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.913289 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.913291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.914098 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.914099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.914161 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.914164 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:55:21.914190 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 02:55:21.914221 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260927 02:55:21.914363 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.910433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.0.98.3:39173)
0927 02:55:21.910842 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:21.910846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:21.910854 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:55:21.910884 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:21.910887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:21.910901 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:21.910970 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:21.911168 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.912187 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.913193 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.913870 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.913937 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.913991 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:21.914259 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 02:55:21.914295 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":64,"threads_started":1}
I20260927 02:55:21.914467 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.910287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39173 (local address 127.0.0.1:53224)
0927 02:55:21.910418 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:21.910454 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:21.910527 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:21.911033 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:21.911034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:21.911035 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:21.911097 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.911098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.912259 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.912260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.913096 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:21.913099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.914027 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.914028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:21.914091 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.914095 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:55:21.914136 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:55:21.914175 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":3,"thread_start_us":43,"threads_started":1}
I20260927 02:55:21.914561 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:21.910411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.0.98.1:46257)
0927 02:55:21.910674 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:21.910679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:21.910687 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:55:21.910725 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:21.910728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:21.910742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:21.910835 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:21.910997 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.912115 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.913361 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:21.914006 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:21.914348 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:21.914417 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:21.914443 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 02:55:21.914484 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
I20260927 02:55:21.915045 15969 tablet_service.cc:1467] Processing CreateTablet for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 (DEFAULT_TABLE table=test-table [id=46f884b72dc04b0589c6b0ebec6b6c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:21.915131 16103 tablet_service.cc:1467] Processing CreateTablet for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 (DEFAULT_TABLE table=test-table [id=46f884b72dc04b0589c6b0ebec6b6c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:21.915392 16103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:21.915393 15969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:21.918438 16184 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Bootstrap starting.
I20260927 02:55:21.918895 16183 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: Bootstrap starting.
I20260927 02:55:21.918996 15835 tablet_service.cc:1467] Processing CreateTablet for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 (DEFAULT_TABLE table=test-table [id=46f884b72dc04b0589c6b0ebec6b6c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:21.919145 16184 tablet_bootstrap.cc:654] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:21.919291 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:21.919512 16183 tablet_bootstrap.cc:654] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:21.919742 16184 log.cc:824] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:21.920027 16183 log.cc:824] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:21.920463 16184 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: No bootstrap required, opened a new log
I20260927 02:55:21.920533 16184 ts_tablet_manager.cc:1397] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:55:21.920715 16183 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: No bootstrap required, opened a new log
I20260927 02:55:21.920776 16183 ts_tablet_manager.cc:1397] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:55:21.922029 16187 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Bootstrap starting.
I20260927 02:55:21.922578 16187 tablet_bootstrap.cc:654] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:21.922533 16184 raft_consensus.cc:348] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:21.922612 16183 raft_consensus.cc:348] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:21.922721 16184 raft_consensus.cc:374] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:21.922808 16183 raft_consensus.cc:374] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:21.922839 16184 raft_consensus.cc:729] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f948ebb84c144a38c3de9116cce83ed, State: Initialized, Role: FOLLOWER
I20260927 02:55:21.922876 16183 raft_consensus.cc:729] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af445c338d341fa85f6c4c8180b0aed, State: Initialized, Role: FOLLOWER
I20260927 02:55:21.922991 16184 consensus_queue.cc:260] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:21.923017 16183 consensus_queue.cc:260] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:21.923136 16187 log.cc:824] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:21.923265 16183 ts_tablet_manager.cc:1428] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:55:21.923265 16184 ts_tablet_manager.cc:1428] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 02:55:21.923385 16169 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260927 02:55:21.923615 16035 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260927 02:55:21.923874 16187 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: No bootstrap required, opened a new log
I20260927 02:55:21.923939 16187 ts_tablet_manager.cc:1397] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:55:21.925307 16187 raft_consensus.cc:348] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:21.925433 16187 raft_consensus.cc:374] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:21.925463 16187 raft_consensus.cc:729] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6001bd73ec4f61963644600e60619d, State: Initialized, Role: FOLLOWER
I20260927 02:55:21.925566 16187 consensus_queue.cc:260] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:21.925861 16187 ts_tablet_manager.cc:1428] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:55:21.925885 15901 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
W20260927 02:55:21.970362 16036 tablet.cc:2369] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:55:22.001492 16188 raft_consensus.cc:482] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:22.001603 16188 raft_consensus.cc:504] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:22.001901 16188 leader_election.cc:288] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173), dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257)
I20260927 02:55:22.004690 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:22.002204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34867 (local address 127.0.98.3:39173)
0927 02:55:22.002255 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:22.002257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:22.002265 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:55:22.002293 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:22.002294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:22.002305 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:22.002358 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:22.002516 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.003071 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.004026 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.004378 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.004435 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:22.004445 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:22.004589 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 02:55:22.004623 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:55:22.004694 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:22.002123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50641 (local address 127.0.98.1:46257)
0927 02:55:22.002187 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:22.002189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:22.002321 (+ 132us) server_negotiation.cc:407] Connection header received
0927 02:55:22.002359 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:22.002360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:22.002372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:22.002431 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:22.002575 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.003048 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.004018 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.004380 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.004437 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:22.004447 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:22.004593 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:55:22.004630 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:55:22.004737 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:22.002063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46257 (local address 127.0.98.2:50641)
0927 02:55:22.002260 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:22.002269 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:22.002314 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:22.002436 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:22.002437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:22.002438 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:22.002510 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:22.002512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.003126 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.003127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:22.003933 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:22.003935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.004448 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.004449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:22.004528 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:22.004532 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:55:22.004578 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:55:22.004619 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":34,"thread_start_us":104,"threads_started":1}
I20260927 02:55:22.004738 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:22.002054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39173 (local address 127.0.98.2:34867)
0927 02:55:22.002105 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:22.002115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:22.002152 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:22.002375 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:22.002377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:22.002379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:22.002449 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:22.002451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.003150 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.003151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:22.003936 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:22.003938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.004446 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.004447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:22.004526 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:22.004529 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:55:22.004573 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:55:22.004622 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":41}
I20260927 02:55:22.005087 16123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c713e4b7d4dc890f9cb2f1c4b98a5" candidate_uuid: "0f948ebb84c144a38c3de9116cce83ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af445c338d341fa85f6c4c8180b0aed" is_pre_election: true
I20260927 02:55:22.005086 15855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c713e4b7d4dc890f9cb2f1c4b98a5" candidate_uuid: "0f948ebb84c144a38c3de9116cce83ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6001bd73ec4f61963644600e60619d" is_pre_election: true
I20260927 02:55:22.005225 15855 raft_consensus.cc:2454] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f948ebb84c144a38c3de9116cce83ed in term 0.
I20260927 02:55:22.005225 16123 raft_consensus.cc:2454] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f948ebb84c144a38c3de9116cce83ed in term 0.
I20260927 02:55:22.005429 15923 leader_election.cc:302] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f948ebb84c144a38c3de9116cce83ed, dd6001bd73ec4f61963644600e60619d; no voters:
I20260927 02:55:22.005546 16188 raft_consensus.cc:2781] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:55:22.005600 16188 raft_consensus.cc:482] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:55:22.005616 16188 raft_consensus.cc:3037] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:22.006273 16188 raft_consensus.cc:504] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:22.006385 16188 leader_election.cc:288] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [CANDIDATE]: Term 1 election: Requested vote from peers 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173), dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257)
I20260927 02:55:22.006512 15855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c713e4b7d4dc890f9cb2f1c4b98a5" candidate_uuid: "0f948ebb84c144a38c3de9116cce83ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6001bd73ec4f61963644600e60619d"
I20260927 02:55:22.006572 16123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c713e4b7d4dc890f9cb2f1c4b98a5" candidate_uuid: "0f948ebb84c144a38c3de9116cce83ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af445c338d341fa85f6c4c8180b0aed"
I20260927 02:55:22.006595 15855 raft_consensus.cc:3037] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:22.006646 16123 raft_consensus.cc:3037] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:22.007284 15855 raft_consensus.cc:2454] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f948ebb84c144a38c3de9116cce83ed in term 1.
I20260927 02:55:22.007367 16123 raft_consensus.cc:2454] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f948ebb84c144a38c3de9116cce83ed in term 1.
I20260927 02:55:22.007439 15923 leader_election.cc:302] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f948ebb84c144a38c3de9116cce83ed, dd6001bd73ec4f61963644600e60619d; no voters:
I20260927 02:55:22.007536 16188 raft_consensus.cc:2781] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:55:22.007684 16188 raft_consensus.cc:686] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 1 LEADER]: Becoming Leader. State: Replica: 0f948ebb84c144a38c3de9116cce83ed, State: Running, Role: LEADER
I20260927 02:55:22.007761 16188 consensus_queue.cc:237] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:22.008546 15709 catalog_manager.cc:5495] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f948ebb84c144a38c3de9116cce83ed (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0f948ebb84c144a38c3de9116cce83ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } health_report { overall_health: HEALTHY } } }
I20260927 02:55:22.027037 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:22.025028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.0.98.2:45319)
0927 02:55:22.025081 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:22.025083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:22.025092 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:55:22.025157 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:22.025157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:22.025174 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:22.025225 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:55:22.025311 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.025755 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.026416 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:22.026598 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:22.026640 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:22.026739 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:55:22.026881 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:55:22.026940 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:22.026957 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 02:55:22.026972 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:55:22.028635 16123 raft_consensus.cc:1261] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 1 FOLLOWER]: Refusing update from remote peer 0f948ebb84c144a38c3de9116cce83ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:55:22.028645 15855 raft_consensus.cc:1261] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Refusing update from remote peer 0f948ebb84c144a38c3de9116cce83ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:55:22.028915 16188 consensus_queue.cc:1035] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:55:22.029044 16193 consensus_queue.cc:1035] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:55:22.031196 16196 mvcc.cc:205] Tried to move back new op lower bound from 7333796749426921472 to 7333796749344264192. Current Snapshot: MvccSnapshot[applied={T|T < 7333796749426921472}]
I20260927 02:55:22.031467 16198 mvcc.cc:205] Tried to move back new op lower bound from 7333796749426921472 to 7333796749344264192. Current Snapshot: MvccSnapshot[applied={T|T < 7333796749426921472}]
I20260927 02:55:22.031764 16197 mvcc.cc:205] Tried to move back new op lower bound from 7333796749426921472 to 7333796749344264192. Current Snapshot: MvccSnapshot[applied={T|T < 7333796749426921472 or (T in {7333796749426921472})}]
I20260927 02:55:22.033823 15710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:55:22.033902 15710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:55:22.034551 15710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Sending DeleteTablet for 3 replicas of tablet 873c713e4b7d4dc890f9cb2f1c4b98a5
I20260927 02:55:22.034900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 15772
I20260927 02:55:22.034915 15969 tablet_service.cc:1514] Processing DeleteTablet for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58782
I20260927 02:55:22.034929 16103 tablet_service.cc:1514] Processing DeleteTablet for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53224
I20260927 02:55:22.035466 16207 tablet_replica.cc:330] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: stopping tablet replica
I20260927 02:55:22.035491 16206 tablet_replica.cc:330] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: stopping tablet replica
I20260927 02:55:22.035555 16206 raft_consensus.cc:2229] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:55:22.035554 16207 raft_consensus.cc:2229] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:55:22.035643 16206 raft_consensus.cc:2258] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:22.035662 16207 raft_consensus.cc:2258] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:22.036011 16206 ts_tablet_manager.cc:1905] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:22.036815 16207 ts_tablet_manager.cc:1905] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:22.037438 16206 ts_tablet_manager.cc:1918] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:22.037496 16206 log.cc:1196] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/873c713e4b7d4dc890f9cb2f1c4b98a5
I20260927 02:55:22.037742 16206 ts_tablet_manager.cc:1939] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 6af445c338d341fa85f6c4c8180b0aed: Deleting consensus metadata
I20260927 02:55:22.038237 15696 catalog_manager.cc:4841] TS 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173): tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 (table test-table [id=46f884b72dc04b0589c6b0ebec6b6c08]) successfully deleted
I20260927 02:55:22.038317 16207 ts_tablet_manager.cc:1918] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:22.038370 16207 log.cc:1196] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/873c713e4b7d4dc890f9cb2f1c4b98a5
I20260927 02:55:22.038607 16207 ts_tablet_manager.cc:1939] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P 0f948ebb84c144a38c3de9116cce83ed: Deleting consensus metadata
I20260927 02:55:22.039053 15695 catalog_manager.cc:4841] TS 0f948ebb84c144a38c3de9116cce83ed (127.0.98.2:45319): tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 (table test-table [id=46f884b72dc04b0589c6b0ebec6b6c08]) successfully deleted
W20260927 02:55:22.041883 15695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46257 (error 108)
I20260927 02:55:22.042121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 15906
I20260927 02:55:22.042667 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:22.042183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46257 (local address 127.0.0.1:51284)
0927 02:55:22.042593 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:22.042635 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389}
W20260927 02:55:22.043015 15695 catalog_manager.cc:4584] TS dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46257: connect: Connection refused (error 111)
I20260927 02:55:22.047098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 16040
I20260927 02:55:22.050587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 15678
I20260927 02:55:42.055035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--webserver_interface=127.0.98.62
--webserver_port=42775
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:55:42.133917 16209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:42.134105 16209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:42.134126 16209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:42.135274 16209 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:55:42.135311 16209 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:55:42.135324 16209 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:55:42.135334 16209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:55:42.135345 16209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:42.135370 16209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:42.135393 16209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:55:42.135413 16209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:55:42.136837 16209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:42.137058 16209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:42.137262 16209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:55:42.145485 16209 server_base.cc:1034] running on GCE node
W20260927 02:55:42.145447 16215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:42.145439 16218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:42.145437 16216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:42.145969 16209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:42.146267 16209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:42.147416 16209 hybrid_clock.cc:648] HybridClock initialized: now 1790477742147398 us; error 33 us; skew 500 ppm
Sep 27 02:55:42 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:42.150005 16209 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:55:42.150444 16209 webserver.cc:460] Webserver started at http://127.0.98.62:42775/ using document root <none> and password file <none>
I20260927 02:55:42.150638 16209 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:42.150696 16209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:42.152614 16209 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:42.153255 16225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.153417 16209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:42.153499 16209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "34e0318197de404aa68c8c0bc1cd1b7a"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "eda5e24939292a449ab4e0df7c10664e"
server_key_iv: "4922fd07ebe8dad9710c5a042a29fbc9"
server_key_version: "encryptionkey@0"
I20260927 02:55:42.153905 16209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:42.162488 16209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:42.164239 16209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:42.164366 16209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:42.168191 16209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45791
I20260927 02:55:42.168226 16277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45791 every 8 connection(s)
I20260927 02:55:42.168566 16209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:55:42.169531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 16209
I20260927 02:55:42.169788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46257
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=40305
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:55:42.170056 16278 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:55:42.170675 16278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Bootstrap starting.
I20260927 02:55:42.179005 16278 log.cc:824] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:42.180773 16278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:55:42.181001 16278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Bootstrap complete.
I20260927 02:55:42.183053 16278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:55:42.183274 16278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e0318197de404aa68c8c0bc1cd1b7a, State: Initialized, Role: FOLLOWER
I20260927 02:55:42.183429 16278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:55:42.183502 16278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:55:42.183555 16278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:55:42.183602 16278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:55:42.184557 16278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:55:42.184648 16278 leader_election.cc:302] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34e0318197de404aa68c8c0bc1cd1b7a; no voters:
I20260927 02:55:42.184800 16278 leader_election.cc:288] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:55:42.184867 16282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:55:42.185037 16278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:55:42.185062 16282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 2 LEADER]: Becoming Leader. State: Replica: 34e0318197de404aa68c8c0bc1cd1b7a, State: Running, Role: LEADER
I20260927 02:55:42.185170 16282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:55:42.185461 16283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260927 02:55:42.185606 16283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:42.185779 16284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34e0318197de404aa68c8c0bc1cd1b7a. Latest consensus state: current_term: 2 leader_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260927 02:55:42.185849 16284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: This master's current role is: LEADER
I20260927 02:55:42.185945 16287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:55:42.186691 16287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:55:42.186882 16287 catalog_manager.cc:1259] Loaded cluster ID: 5564eb47483a4de2b0202bf057665d15
I20260927 02:55:42.187086 16287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:55:42.188257 16287 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:55:42.188453 16287 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Loaded TSK: 0
I20260927 02:55:42.201288 16287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Generated new TSK 1
I20260927 02:55:42.201385 16287 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:55:42.253535 16280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:42.253726 16280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:42.253746 16280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:42.254895 16280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:42.254933 16280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:42.254958 16280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:55:42.256363 16280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40305
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:42.256564 16280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:42.256760 16280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:42.262403 16308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:42.262435 16305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:42.262485 16280 server_base.cc:1034] running on GCE node
W20260927 02:55:42.262620 16306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:42.262844 16280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:42.263113 16280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:42.264282 16280 hybrid_clock.cc:648] HybridClock initialized: now 1790477742264269 us; error 27 us; skew 500 ppm
Sep 27 02:55:42 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:42.267110 16280 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:55:42.267539 16280 webserver.cc:460] Webserver started at http://127.0.98.1:40305/ using document root <none> and password file <none>
I20260927 02:55:42.267727 16280 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:42.267786 16280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:42.269085 16280 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:42.269711 16315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.269858 16280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:55:42.269933 16280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "dd6001bd73ec4f61963644600e60619d"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "149fb10e898049a6135512fee543e908"
server_key_iv: "e58a3e0f2274a8c3754c17db185deac2"
server_key_version: "encryptionkey@0"
I20260927 02:55:42.270238 16280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:42.291515 16280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:42.293664 16280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:42.293802 16280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:42.294024 16280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:42.294489 16322 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:55:42.296242 16280 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:55:42.296288 16280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:55:42.296314 16280 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:55:42.296938 16280 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:55:42.296970 16280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:55:42.297049 16322 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Bootstrap starting.
I20260927 02:55:42.303428 16280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46257
I20260927 02:55:42.303486 16429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46257 every 8 connection(s)
I20260927 02:55:42.303821 16280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:55:42.304347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 16280
I20260927 02:55:42.304594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45319
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=44497
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:55:42.307611 16322 log.cc:824] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:42.309953 16322 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:55:42.310642 16322 tablet_bootstrap.cc:492] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Bootstrap complete.
Sep 27 02:55:42 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:42.311635 16322 ts_tablet_manager.cc:1397] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260927 02:55:42.313341 16322 raft_consensus.cc:348] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:42.313750 16322 raft_consensus.cc:729] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6001bd73ec4f61963644600e60619d, State: Initialized, Role: FOLLOWER
I20260927 02:55:42.313936 16322 consensus_queue.cc:260] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260927 02:55:42.314206 16322 ts_tablet_manager.cc:1428] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 02:55:42.315665 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.304519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:46471)
0927 02:55:42.305198 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.305241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:42.305784 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:42.306356 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:42.306360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:42.306704 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:42.306990 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.306997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.308434 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.308434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.309182 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.309184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.309723 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.309725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.309875 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.310410 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:42.310419 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:42.312097 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:42.314197 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:42.314201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:42.314203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:42.314457 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:42.314722 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:42.314722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:42.314722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:42.314846 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:42.315254 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:42.315256 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:42.315357 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 02:55:42.315426 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:55:42.315538 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":78,"threads_started":1}
I20260927 02:55:42.315835 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.305013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46471 (local address 127.0.98.62:45791)
0927 02:55:42.305994 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.306003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.306037 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:55:42.306150 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.306154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.306192 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.306305 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:42.307118 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.308309 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.309261 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.309634 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.309731 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.312218 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:42.312221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:42.312222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:42.312232 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:42.314057 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.314536 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.314540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.314540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.314655 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.314905 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.314906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.314906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.315189 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:42.315277 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.315516 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.315649 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":66,"threads_started":1}
I20260927 02:55:42.316459 16430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260927 02:55:42.316552 16430 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:42.316721 16430 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260927 02:55:42.317345 16242 ts_manager.cc:194] Registered new tserver with Master: dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257)
I20260927 02:55:42.318279 16242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46471
I20260927 02:55:42.319156 16430 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
Sep 27 02:55:42 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:55:42.325189 16438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.317923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.98.1:46257)
0927 02:55:42.318054 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.318058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.318146 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:55:42.318240 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.318242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.318258 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.318323 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:42.318744 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.319702 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.320734 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.321001 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.321050 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.322842 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:42.322849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:42.322850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:42.322856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:42.324146 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.324468 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.324470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.324471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.324521 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.324677 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.324678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.324678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.324803 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:42.324868 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.325028 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.325106 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260927 02:55:42.325191 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.317861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46257 (local address 127.0.0.1:55552)
0927 02:55:42.318057 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.318087 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:42.318153 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:42.318331 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:42.318333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:42.318627 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:42.318677 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.318678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.319761 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.319762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.320666 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.320668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.321059 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.321061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.321129 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.321491 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:42.321495 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:42.322777 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:42.324219 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:42.324225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:42.324226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:42.324401 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:42.324574 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:42.324575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:42.324575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:42.324629 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:42.324859 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:42.324865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:42.324920 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 02:55:42.324989 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:55:42.325094 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260927 02:55:42.325528 16364 tablet_service.cc:1514] Processing DeleteTablet for tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55552
I20260927 02:55:42.325934 16439 tablet_replica.cc:330] stopping tablet replica
I20260927 02:55:42.326033 16439 raft_consensus.cc:2229] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:55:42.326090 16439 raft_consensus.cc:2258] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:42.326385 16439 ts_tablet_manager.cc:1905] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:42.328749 16439 ts_tablet_manager.cc:1918] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:42.328817 16439 log.cc:1196] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal/wals/873c713e4b7d4dc890f9cb2f1c4b98a5
I20260927 02:55:42.329015 16439 ts_tablet_manager.cc:1939] T 873c713e4b7d4dc890f9cb2f1c4b98a5 P dd6001bd73ec4f61963644600e60619d: Deleting consensus metadata
I20260927 02:55:42.329460 16227 catalog_manager.cc:4841] TS dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257): tablet 873c713e4b7d4dc890f9cb2f1c4b98a5 (table test-table [id=46f884b72dc04b0589c6b0ebec6b6c08]) successfully deleted
W20260927 02:55:42.389649 16434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:42.389832 16434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:42.389859 16434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:42.391018 16434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:42.391054 16434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:42.391080 16434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:55:42.392434 16434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44497
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:42.392619 16434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:42.392813 16434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:42.397373 16444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:42.397382 16445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:42.397539 16447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:42.397575 16434 server_base.cc:1034] running on GCE node
I20260927 02:55:42.397765 16434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:42.397985 16434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:42.399116 16434 hybrid_clock.cc:648] HybridClock initialized: now 1790477742399097 us; error 29 us; skew 500 ppm
Sep 27 02:55:42 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:42.401472 16434 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:55:42.401866 16434 webserver.cc:460] Webserver started at http://127.0.98.2:44497/ using document root <none> and password file <none>
I20260927 02:55:42.402045 16434 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:42.402102 16434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:42.403380 16434 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:55:42.403873 16454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.404016 16434 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.404088 16434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "0f948ebb84c144a38c3de9116cce83ed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "c487e51baa1c44d869291073e6ab8ae3"
server_key_iv: "3e2c6a99c300bde59e643ff76704ba04"
server_key_version: "encryptionkey@0"
I20260927 02:55:42.404388 16434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:42.422333 16434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:42.424345 16434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:42.424467 16434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:42.424659 16434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:42.424925 16434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:42.424958 16434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.424984 16434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:42.425009 16434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.430227 16434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45319
I20260927 02:55:42.430287 16567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45319 every 8 connection(s)
I20260927 02:55:42.430584 16434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:55:42.432672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 16434
I20260927 02:55:42.432940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39173
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=38549
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:55:42 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:42.439790 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.431233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50507 (local address 127.0.98.62:45791)
0927 02:55:42.431305 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.431307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.431455 (+ 148us) server_negotiation.cc:407] Connection header received
0927 02:55:42.431947 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.431948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.431961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.432010 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:42.433046 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.433548 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.434679 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.435000 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.435059 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.437194 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:42.437195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:42.437196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:42.437202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:42.438450 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.438921 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.438922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.438922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.438971 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.439218 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.439218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.439218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.439320 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:42.439364 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.439658 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.439723 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:55:42.440027 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.431213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:50507)
0927 02:55:42.431344 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.431382 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:42.431839 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:42.432065 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:42.432068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:42.432668 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:42.432946 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.432954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.433634 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.433635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.434600 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.434603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.435086 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.435087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.435188 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.435627 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:42.435634 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:42.437083 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:42.438526 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:42.438533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:42.438535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:42.438857 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:42.439030 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:42.439031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:42.439032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:42.439163 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:42.439373 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:42.439379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:42.439534 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 02:55:42.439650 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 02:55:42.439808 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260927 02:55:42.440513 16568 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260927 02:55:42.440610 16568 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:42.440774 16568 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260927 02:55:42.441169 16242 ts_manager.cc:194] Registered new tserver with Master: 0f948ebb84c144a38c3de9116cce83ed (127.0.98.2:45319)
I20260927 02:55:42.441514 16242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50507
W20260927 02:55:42.511651 16571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:55:42.511847 16571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:55:42.511878 16571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:55:42.512964 16571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:55:42.513000 16571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:55:42.513036 16571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:55:42.514374 16571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38549
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:55:42.514590 16571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:55:42.514801 16571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:55:42.519425 16579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:42.519428 16576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:55:42.519464 16577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:55:42.519625 16571 server_base.cc:1034] running on GCE node
I20260927 02:55:42.519829 16571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:55:42.520030 16571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:55:42.521176 16571 hybrid_clock.cc:648] HybridClock initialized: now 1790477742521163 us; error 28 us; skew 500 ppm
Sep 27 02:55:42 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:55:42.523505 16571 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:55:42.523890 16571 webserver.cc:460] Webserver started at http://127.0.98.3:38549/ using document root <none> and password file <none>
I20260927 02:55:42.524068 16571 fs_manager.cc:359] Metadata directory not provided
I20260927 02:55:42.524124 16571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:55:42.525591 16571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:55:42.526193 16586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.526381 16571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:55:42.526460 16571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "6af445c338d341fa85f6c4c8180b0aed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "670870ae6a24400b0c35f7c94a4e77db"
server_key_iv: "521cdb16902c6d51426337ce88d686a6"
server_key_version: "encryptionkey@0"
I20260927 02:55:42.526793 16571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:55:42.547988 16571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:55:42.549717 16571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:55:42.549847 16571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:55:42.550055 16571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:55:42.550338 16571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:55:42.550367 16571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.550396 16571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:55:42.550417 16571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.555763 16571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39173
I20260927 02:55:42.555815 16699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39173 every 8 connection(s)
I20260927 02:55:42.556159 16571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:55:42.557601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 16571
Sep 27 02:55:42 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:55:42.565052 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.558004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.0.98.62:45791)
0927 02:55:42.558119 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.558121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.558201 (+ 80us) server_negotiation.cc:407] Connection header received
0927 02:55:42.558271 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.558271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.558280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.558332 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:42.558991 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.559802 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.560924 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.561129 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.561192 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.562494 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:42.562495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:42.562495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:42.562501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:42.563978 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.564293 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.564294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.564295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.564354 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.564559 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.564560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.564560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.564686 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:42.564843 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.564881 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.564951 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260927 02:55:42.566534 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.556777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52449 (local address 127.0.98.62:45791)
0927 02:55:42.556818 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.556820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.557117 (+ 297us) server_negotiation.cc:407] Connection header received
0927 02:55:42.557672 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.557673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.557685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.557740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:42.558982 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.559795 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.561195 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.561385 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.561439 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.563747 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:42.563749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:42.563749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:42.563755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:42.565229 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.565652 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.565653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.565653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.565708 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.565957 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.565958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.565959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.566071 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:42.566124 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.566382 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.566454 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":17}
I20260927 02:55:42.566709 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.556782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:52449)
0927 02:55:42.556988 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.557023 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:42.557567 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:42.557833 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:42.557837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:42.558559 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:55:42.558863 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.558870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.559910 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.559911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.561120 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.561122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.561458 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.561459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.561557 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.561922 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:55:42.561932 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:55:42.563659 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:55:42.565313 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:42.565320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:42.565322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:42.565593 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:42.565766 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:55:42.565766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:55:42.565767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:55:42.565898 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:55:42.566129 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:55:42.566136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:55:42.566262 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:55:42.566345 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:55:42.566492 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":133,"threads_started":1}
I20260927 02:55:42.567149 16700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260927 02:55:42.567229 16700 heartbeater.cc:464] Registering TS with master...
I20260927 02:55:42.567374 16700 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260927 02:55:42.567663 16242 ts_manager.cc:194] Registered new tserver with Master: 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173)
I20260927 02:55:42.567979 16242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52449
I20260927 02:55:42.576534 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:55:42.579272 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.576866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.0.98.62:45791)
0927 02:55:42.576920 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.576921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.577075 (+ 154us) server_negotiation.cc:407] Connection header received
0927 02:55:42.577128 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.577128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.577163 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.577219 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:55:42.577303 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.577776 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.578528 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.578710 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.578793 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.579000 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:55:42.579046 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 02:55:42.579208 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60076: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 02:55:42.579316 16433 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60076: token expired
W20260927 02:55:42.579427 16174 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:45791)
I20260927 02:55:42.584748 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.579643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.0.98.62:45791)
0927 02:55:42.579682 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.579684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.579692 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:55:42.579711 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.579712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.579719 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.579762 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:55:42.580179 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.580648 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.581306 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.581485 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.581535 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.582611 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:55:42.582612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:55:42.582613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:55:42.582618 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:55:42.583899 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.584201 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.584202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.584203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.584262 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:55:42.584409 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:55:42.584409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:55:42.584409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:55:42.584499 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:55:42.584572 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.584634 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.584691 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:55:42.585821 16242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:55:42.587477 16242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:55:42.589600 16364 tablet_service.cc:1467] Processing CreateTablet for tablet 561d9341ac464772b6d7aef937194a52 (DEFAULT_TABLE table=test-table [id=eea79583a1a64f9592e7c3e946b57c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:42.589771 16364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561d9341ac464772b6d7aef937194a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:42.591102 16322 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Bootstrap starting.
I20260927 02:55:42.591370 16322 tablet_bootstrap.cc:654] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:42.592016 16322 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: No bootstrap required, opened a new log
I20260927 02:55:42.592056 16322 ts_tablet_manager.cc:1397] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:55:42.592216 16322 raft_consensus.cc:348] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.592269 16322 raft_consensus.cc:374] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:42.592312 16322 raft_consensus.cc:729] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6001bd73ec4f61963644600e60619d, State: Initialized, Role: FOLLOWER
I20260927 02:55:42.592360 16322 consensus_queue.cc:260] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.592447 16322 ts_tablet_manager.cc:1428] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:55:42.593469 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.589213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:40524)
0927 02:55:42.589383 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.589392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:42.589435 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:42.589823 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:42.589823 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:42.589825 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:42.589853 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.589854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.591163 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.591164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.592411 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.592414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.593270 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.593271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.593319 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.593334 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 02:55:42.593376 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:55:42.593406 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":46,"thread_start_us":100,"threads_started":1}
I20260927 02:55:42.593513 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.589430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.0.98.2:45319)
0927 02:55:42.589569 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.589575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.589585 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:55:42.589627 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.589629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.589645 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.589724 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:42.589924 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.591082 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.592502 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.593199 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.593271 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.593328 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.593389 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.593434 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260927 02:55:42.593675 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.589181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39173 (local address 127.0.0.1:44062)
0927 02:55:42.589286 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.589296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:42.589343 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:42.589770 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:42.589771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:42.589772 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:42.589812 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.589814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.591152 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.591153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.592411 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.592414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.593463 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.593464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.593525 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.593529 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:55:42.593581 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:55:42.593618 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":74}
I20260927 02:55:42.593887 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.589290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.0.98.3:39173)
0927 02:55:42.589609 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.589612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.589620 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:55:42.589652 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.589654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.589670 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.589882 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:42.589967 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.591052 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.592504 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.593392 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.593452 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.593504 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.593769 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.593810 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
I20260927 02:55:42.594208 16502 tablet_service.cc:1467] Processing CreateTablet for tablet 561d9341ac464772b6d7aef937194a52 (DEFAULT_TABLE table=test-table [id=eea79583a1a64f9592e7c3e946b57c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:42.594455 16634 tablet_service.cc:1467] Processing CreateTablet for tablet 561d9341ac464772b6d7aef937194a52 (DEFAULT_TABLE table=test-table [id=eea79583a1a64f9592e7c3e946b57c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:55:42.594537 16502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561d9341ac464772b6d7aef937194a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:42.594710 16634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561d9341ac464772b6d7aef937194a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:55:42.597220 16711 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Bootstrap starting.
I20260927 02:55:42.597147 16710 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Bootstrap starting.
I20260927 02:55:42.597775 16710 tablet_bootstrap.cc:654] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:42.597816 16711 tablet_bootstrap.cc:654] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Neither blocks nor log segments found. Creating new log.
I20260927 02:55:42.598276 16711 log.cc:824] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:42.598281 16710 log.cc:824] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:55:42.598937 16710 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: No bootstrap required, opened a new log
I20260927 02:55:42.598942 16711 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: No bootstrap required, opened a new log
I20260927 02:55:42.598989 16710 ts_tablet_manager.cc:1397] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:55:42.598992 16711 ts_tablet_manager.cc:1397] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:55:42.600322 16710 raft_consensus.cc:348] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.600302 16711 raft_consensus.cc:348] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.600455 16710 raft_consensus.cc:374] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:42.600455 16711 raft_consensus.cc:374] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:55:42.600476 16710 raft_consensus.cc:729] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f948ebb84c144a38c3de9116cce83ed, State: Initialized, Role: FOLLOWER
I20260927 02:55:42.600486 16711 raft_consensus.cc:729] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af445c338d341fa85f6c4c8180b0aed, State: Initialized, Role: FOLLOWER
I20260927 02:55:42.600587 16710 consensus_queue.cc:260] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.600608 16711 consensus_queue.cc:260] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.600849 16568 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260927 02:55:42.600876 16700 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260927 02:55:42.600847 16710 ts_tablet_manager.cc:1428] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:55:42.601122 16711 ts_tablet_manager.cc:1428] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:55:42.638093 16714 raft_consensus.cc:482] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:55:42.638213 16714 raft_consensus.cc:504] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.638550 16714 leader_election.cc:288] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173), dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257)
I20260927 02:55:42.641219 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.638706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39173 (local address 127.0.98.2:52091)
0927 02:55:42.638792 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.638804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:42.638863 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:42.638995 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:42.638996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:42.638998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:42.639066 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.639067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.639669 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.639670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.640434 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.640436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.640982 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.640982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.641061 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.641064 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:55:42.641107 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:55:42.641145 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 02:55:42.641219 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.638773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52091 (local address 127.0.98.3:39173)
0927 02:55:42.638814 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.638816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.638848 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:55:42.638925 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.638926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.638937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.638985 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:42.639124 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.639598 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.640525 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.640906 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.640966 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.640977 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.641122 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.641159 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:55:42.641319 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.638727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46257 (local address 127.0.98.2:50635)
0927 02:55:42.638902 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.638911 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:55:42.638969 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:55:42.639112 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:55:42.639113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:55:42.639114 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:55:42.639176 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.639177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.639786 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.639787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.640491 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:55:42.640492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.641087 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.641088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:55:42.641151 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.641156 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:55:42.641208 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:55:42.641243 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":125,"threads_started":1}
I20260927 02:55:42.641315 16438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.638792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50635 (local address 127.0.98.1:46257)
0927 02:55:42.638836 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.638838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.638972 (+ 134us) server_negotiation.cc:407] Connection header received
0927 02:55:42.639021 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.639022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.639045 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.639101 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:55:42.639231 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.639714 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.640583 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.641009 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.641070 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.641151 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.641216 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.641256 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:55:42.641561 16654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561d9341ac464772b6d7aef937194a52" candidate_uuid: "0f948ebb84c144a38c3de9116cce83ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af445c338d341fa85f6c4c8180b0aed" is_pre_election: true
I20260927 02:55:42.641589 16384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561d9341ac464772b6d7aef937194a52" candidate_uuid: "0f948ebb84c144a38c3de9116cce83ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6001bd73ec4f61963644600e60619d" is_pre_election: true
I20260927 02:55:42.641701 16654 raft_consensus.cc:2454] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f948ebb84c144a38c3de9116cce83ed in term 0.
I20260927 02:55:42.641724 16384 raft_consensus.cc:2454] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f948ebb84c144a38c3de9116cce83ed in term 0.
I20260927 02:55:42.641906 16457 leader_election.cc:302] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f948ebb84c144a38c3de9116cce83ed, 6af445c338d341fa85f6c4c8180b0aed; no voters:
I20260927 02:55:42.642050 16714 raft_consensus.cc:2781] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:55:42.642094 16714 raft_consensus.cc:482] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:55:42.642117 16714 raft_consensus.cc:3037] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:42.642820 16714 raft_consensus.cc:504] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.642936 16714 leader_election.cc:288] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [CANDIDATE]: Term 1 election: Requested vote from peers 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173), dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257)
I20260927 02:55:42.643079 16654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561d9341ac464772b6d7aef937194a52" candidate_uuid: "0f948ebb84c144a38c3de9116cce83ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af445c338d341fa85f6c4c8180b0aed"
I20260927 02:55:42.643097 16384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561d9341ac464772b6d7aef937194a52" candidate_uuid: "0f948ebb84c144a38c3de9116cce83ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6001bd73ec4f61963644600e60619d"
I20260927 02:55:42.643158 16654 raft_consensus.cc:3037] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:42.643167 16384 raft_consensus.cc:3037] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:55:42.643826 16654 raft_consensus.cc:2454] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f948ebb84c144a38c3de9116cce83ed in term 1.
I20260927 02:55:42.643885 16384 raft_consensus.cc:2454] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f948ebb84c144a38c3de9116cce83ed in term 1.
I20260927 02:55:42.644009 16457 leader_election.cc:302] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f948ebb84c144a38c3de9116cce83ed, 6af445c338d341fa85f6c4c8180b0aed; no voters:
I20260927 02:55:42.644094 16714 raft_consensus.cc:2781] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:55:42.644244 16714 raft_consensus.cc:686] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 LEADER]: Becoming Leader. State: Replica: 0f948ebb84c144a38c3de9116cce83ed, State: Running, Role: LEADER
I20260927 02:55:42.644316 16714 consensus_queue.cc:237] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:55:42.645139 16240 catalog_manager.cc:5495] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f948ebb84c144a38c3de9116cce83ed (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0f948ebb84c144a38c3de9116cce83ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } health_report { overall_health: UNKNOWN } } }
I20260927 02:55:42.651480 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.649512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.0.98.2:45319)
0927 02:55:42.649548 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:55:42.649550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:55:42.649559 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:55:42.649589 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:55:42.649590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:55:42.649604 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:55:42.649660 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:55:42.649737 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.650183 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.650865 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:55:42.651052 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:55:42.651117 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:55:42.651215 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:55:42.651304 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:55:42.651364 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:55:42.651396 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 02:55:42.651413 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:55:42.653029 16384 raft_consensus.cc:1261] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Refusing update from remote peer 0f948ebb84c144a38c3de9116cce83ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:55:42.653030 16654 raft_consensus.cc:1261] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 1 FOLLOWER]: Refusing update from remote peer 0f948ebb84c144a38c3de9116cce83ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:55:42.653299 16717 consensus_queue.cc:1035] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:55:42.653426 16714 consensus_queue.cc:1035] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:55:42.657748 16241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:55:42.657827 16241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:55:42.658469 16241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Sending DeleteTablet for 3 replicas of tablet 561d9341ac464772b6d7aef937194a52
I20260927 02:55:42.658793 16502 tablet_service.cc:1514] Processing DeleteTablet for tablet 561d9341ac464772b6d7aef937194a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40524
I20260927 02:55:42.658828 16364 tablet_service.cc:1514] Processing DeleteTablet for tablet 561d9341ac464772b6d7aef937194a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55552
I20260927 02:55:42.658921 16439 tablet_replica.cc:330] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: stopping tablet replica
I20260927 02:55:42.658968 16439 raft_consensus.cc:2229] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:55:42.659025 16439 raft_consensus.cc:2258] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:42.658744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 16280
I20260927 02:55:42.659112 16634 tablet_service.cc:1514] Processing DeleteTablet for tablet 561d9341ac464772b6d7aef937194a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:55:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44062
I20260927 02:55:42.659584 16729 tablet_replica.cc:330] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: stopping tablet replica
I20260927 02:55:42.659672 16729 raft_consensus.cc:2229] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:55:42.659761 16729 raft_consensus.cc:2258] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:42.660120 16729 ts_tablet_manager.cc:1905] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:55:42.661542 16729 ts_tablet_manager.cc:1918] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:55:42.661600 16729 log.cc:1196] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/561d9341ac464772b6d7aef937194a52
I20260927 02:55:42.661816 16729 ts_tablet_manager.cc:1939] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Deleting consensus metadata
I20260927 02:55:42.662300 16228 catalog_manager.cc:4841] TS 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173): tablet 561d9341ac464772b6d7aef937194a52 (table test-table [id=eea79583a1a64f9592e7c3e946b57c18]) successfully deleted
W20260927 02:55:42.666301 16227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46257 (error 108)
I20260927 02:55:42.666514 16728 tablet_replica.cc:330] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: stopping tablet replica
I20260927 02:55:42.666622 16728 raft_consensus.cc:2229] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:55:42.666721 16728 raft_consensus.cc:2258] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:55:42.666810 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.666665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46257 (local address 127.0.0.1:55564)
0927 02:55:42.666711 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.666743 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:55:42.666917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 16434
W20260927 02:55:42.666931 16227 catalog_manager.cc:4584] TS dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 561d9341ac464772b6d7aef937194a52: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46257: connect: Connection refused (error 111)
I20260927 02:55:42.671116 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:55:42.671009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:40556)
0927 02:55:42.671060 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:55:42.671086 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:55:42.671160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 16571
I20260927 02:55:42.674705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 16209
I20260927 02:56:02.679011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--webserver_interface=127.0.98.62
--webserver_port=42775
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:02.755795 16731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:02.756000 16731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:02.756031 16731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:02.757169 16731 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:56:02.757205 16731 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:56:02.757222 16731 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:56:02.757236 16731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:02.757252 16731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:02.757269 16731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:02.757289 16731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:02.757309 16731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:02.758703 16731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:02.758955 16731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:02.759136 16731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:02.763458 16737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:02.763513 16739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:02.763490 16736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:02.763619 16731 server_base.cc:1034] running on GCE node
I20260927 02:56:02.763911 16731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:02.764140 16731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:02.765280 16731 hybrid_clock.cc:648] HybridClock initialized: now 1790477762765262 us; error 35 us; skew 500 ppm
Sep 27 02:56:02 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:02.767547 16731 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:02.767962 16731 webserver.cc:460] Webserver started at http://127.0.98.62:42775/ using document root <none> and password file <none>
I20260927 02:56:02.768155 16731 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:02.768198 16731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:02.769414 16731 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:56:02.770021 16746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:02.770157 16731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:02.770231 16731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "34e0318197de404aa68c8c0bc1cd1b7a"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "eda5e24939292a449ab4e0df7c10664e"
server_key_iv: "4922fd07ebe8dad9710c5a042a29fbc9"
server_key_version: "encryptionkey@0"
I20260927 02:56:02.770519 16731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:02.788040 16731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:02.790145 16731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:02.790288 16731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:02.794188 16731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45791
I20260927 02:56:02.794214 16798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45791 every 8 connection(s)
I20260927 02:56:02.794566 16731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:02.795900 16799 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:56:02.796504 16799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Bootstrap starting.
I20260927 02:56:02.803232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 16731
I20260927 02:56:02.803474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46257
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=40305
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:56:02.804459 16799 log.cc:824] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:02.806677 16799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:56:02.806986 16799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Bootstrap complete.
I20260927 02:56:02.808920 16799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:56:02.809126 16799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e0318197de404aa68c8c0bc1cd1b7a, State: Initialized, Role: FOLLOWER
I20260927 02:56:02.809283 16799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:56:02.809356 16799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:02.809404 16799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:02.809449 16799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:56:02.810357 16799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:56:02.810446 16799 leader_election.cc:302] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34e0318197de404aa68c8c0bc1cd1b7a; no voters:
I20260927 02:56:02.810596 16799 leader_election.cc:288] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:56:02.810644 16803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:56:02.810830 16803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [term 3 LEADER]: Becoming Leader. State: Replica: 34e0318197de404aa68c8c0bc1cd1b7a, State: Running, Role: LEADER
I20260927 02:56:02.810868 16799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:02.810977 16803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260927 02:56:02.811278 16804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260927 02:56:02.811347 16804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:02.811543 16805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34e0318197de404aa68c8c0bc1cd1b7a. Latest consensus state: current_term: 3 leader_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e0318197de404aa68c8c0bc1cd1b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260927 02:56:02.811609 16805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:02.811707 16809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:02.812457 16809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:02.812552 16809 catalog_manager.cc:1259] Loaded cluster ID: 5564eb47483a4de2b0202bf057665d15
I20260927 02:56:02.812625 16809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:02.813758 16809 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:02.813925 16809 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Loaded TSK: 0
I20260927 02:56:02.813969 16809 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34e0318197de404aa68c8c0bc1cd1b7a: Loaded TSK: 1
F20260927 02:56:02.814183 16809 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f189e4899ad google::LogMessage::Fail() at ??:0
@ 0x7f189e48c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f189e4894de google::LogMessage::Flush() at ??:0
@ 0x7f189e48aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f18a0bf6104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f18a0bf3d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f18a63e71cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f18a63e748e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f18a63e7594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f18a63adc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f18a63b7ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f18a03570d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f18a034c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f18a11b06db start_thread at ??:0
@ 0x7f189cf8761f clone at ??:0
W20260927 02:56:02.907680 16801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:02.907920 16801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:02.907954 16801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:02.909559 16801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:02.909615 16801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:02.909653 16801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:56:02.911722 16801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40305
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:02.911947 16801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:02.912176 16801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:02.917290 16841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:02.917455 16801 server_base.cc:1034] running on GCE node
W20260927 02:56:02.917508 16840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:02.917292 16843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:02.917770 16801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:02.918027 16801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:02.919198 16801 hybrid_clock.cc:648] HybridClock initialized: now 1790477762919183 us; error 25 us; skew 500 ppm
Sep 27 02:56:02 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:02.921936 16801 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:56:02.922297 16801 webserver.cc:460] Webserver started at http://127.0.98.1:40305/ using document root <none> and password file <none>
I20260927 02:56:02.922513 16801 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:02.922554 16801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:02.923837 16801 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:56:02.924439 16850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:02.924603 16801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:02.924674 16801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "dd6001bd73ec4f61963644600e60619d"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "149fb10e898049a6135512fee543e908"
server_key_iv: "e58a3e0f2274a8c3754c17db185deac2"
server_key_version: "encryptionkey@0"
I20260927 02:56:02.925000 16801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:02.943565 16801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:02.945817 16801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:02.945940 16801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:02.946130 16801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:02.946518 16857 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:56:02.948396 16801 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:56:02.948449 16801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:56:02.948482 16801 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:56:02.949084 16801 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:56:02.949116 16801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:56:02.949203 16857 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Bootstrap starting.
I20260927 02:56:02.955397 16801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46257
I20260927 02:56:02.955499 16964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46257 every 8 connection(s)
I20260927 02:56:02.955782 16801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:56:02.958123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 16801
I20260927 02:56:02.958359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45319
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=44497
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:56:02.959575 16857 log.cc:824] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:02.962141 16857 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:56:02.962421 16857 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Bootstrap complete.
I20260927 02:56:02.963557 16857 ts_tablet_manager.cc:1397] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260927 02:56:02.964941 16857 raft_consensus.cc:348] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:02.965301 16857 raft_consensus.cc:729] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6001bd73ec4f61963644600e60619d, State: Initialized, Role: FOLLOWER
I20260927 02:56:02.965495 16857 consensus_queue.cc:260] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:02.965785 16857 ts_tablet_manager.cc:1428] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260927 02:56:03.012548 16967 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:02.956436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:36227)
0927 02:56:02.956587 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:02.956627 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:02.957029 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:03.012371 (+ 55342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260927 02:56:03.012797 16965 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45791: Client connection negotiation failed: client connection to 127.0.98.62:45791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:56:03.013113 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.013013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:35367)
0927 02:56:03.013058 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.013094 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:56:03.013377 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.013308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:36257)
0927 02:56:03.013342 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.013362 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:56:03.013484 16965 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:56:03.039690 16968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:03.039880 16968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:03.039902 16968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:03.040992 16968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:03.041023 16968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:03.041046 16968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:56:03.042399 16968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44497
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:03.042572 16968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:03.042737 16968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:56:03.047356 16968 server_base.cc:1034] running on GCE node
W20260927 02:56:03.047303 16976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:03.047365 16978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:03.047312 16975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:03.047755 16968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:03.047986 16968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:03.049142 16968 hybrid_clock.cc:648] HybridClock initialized: now 1790477763049125 us; error 26 us; skew 500 ppm
Sep 27 02:56:03 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:03.051447 16968 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:56:03.051877 16968 webserver.cc:460] Webserver started at http://127.0.98.2:44497/ using document root <none> and password file <none>
I20260927 02:56:03.052073 16968 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:03.052119 16968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:03.053294 16968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:56:03.053841 16985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:03.053982 16968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:03.054056 16968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "0f948ebb84c144a38c3de9116cce83ed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "c487e51baa1c44d869291073e6ab8ae3"
server_key_iv: "3e2c6a99c300bde59e643ff76704ba04"
server_key_version: "encryptionkey@0"
I20260927 02:56:03.054368 16968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:03.065570 16968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:03.067610 16968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:03.067730 16968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:03.067912 16968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:03.068252 16992 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:56:03.069547 16968 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:56:03.069583 16968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260927 02:56:03.069619 16968 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:56:03.070186 16968 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:56:03.070217 16968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:56:03.070325 16992 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Bootstrap starting.
I20260927 02:56:03.076509 16968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45319
I20260927 02:56:03.076593 17099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45319 every 8 connection(s)
I20260927 02:56:03.076874 16968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:56:03.077829 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.077467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:42865)
0927 02:56:03.077644 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.077705 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
W20260927 02:56:03.077996 17100 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45791: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
I20260927 02:56:03.078256 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.078139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:57023)
0927 02:56:03.078191 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.078229 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:56:03.078627 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.078521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:34089)
0927 02:56:03.078567 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.078602 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:56:03.078790 17100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:56:03.079226 16992 log.cc:824] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:03.081065 16992 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:56:03.081270 16992 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Bootstrap complete.
I20260927 02:56:03.081957 16992 ts_tablet_manager.cc:1397] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260927 02:56:03.083103 16992 raft_consensus.cc:348] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:03.083348 16992 raft_consensus.cc:729] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f948ebb84c144a38c3de9116cce83ed, State: Initialized, Role: FOLLOWER
I20260927 02:56:03.083448 16992 consensus_queue.cc:260] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:03.083662 16992 ts_tablet_manager.cc:1428] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260927 02:56:03.083896 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.083819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:45203)
0927 02:56:03.083854 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.083879 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:56:03.084132 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.084069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:54481)
0927 02:56:03.084100 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.084119 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:56:03.084793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 16968
I20260927 02:56:03.085011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39173
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=38549
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:03.162681 17105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:03.162958 17105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:03.162992 17105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:03.164588 17105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:03.164635 17105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:03.164671 17105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:56:03.166495 17105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38549
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:03.166725 17105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:03.167006 17105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:03.171686 17113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:03.171705 17110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:03.171898 17111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:03.171734 17105 server_base.cc:1034] running on GCE node
I20260927 02:56:03.172075 17105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:03.172286 17105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:03.173420 17105 hybrid_clock.cc:648] HybridClock initialized: now 1790477763173409 us; error 22 us; skew 500 ppm
Sep 27 02:56:03 dist-test-slave-mfx9 krb5kdc[15665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:03.175711 17105 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:56:03.176071 17105 webserver.cc:460] Webserver started at http://127.0.98.3:38549/ using document root <none> and password file <none>
I20260927 02:56:03.176246 17105 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:03.176303 17105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:03.177464 17105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:56:03.177956 17120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:03.178097 17105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:03.178170 17105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "6af445c338d341fa85f6c4c8180b0aed"
format_stamp: "Formatted at 2026-09-27 02:55:21 on dist-test-slave-mfx9"
server_key: "670870ae6a24400b0c35f7c94a4e77db"
server_key_iv: "521cdb16902c6d51426337ce88d686a6"
server_key_version: "encryptionkey@0"
I20260927 02:56:03.178460 17105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:03.198565 17105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:03.200577 17105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:03.200706 17105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:03.200910 17105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:03.201179 17105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:03.201210 17105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:03.201236 17105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:03.201270 17105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:03.206367 17105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39173
I20260927 02:56:03.206419 17233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39173 every 8 connection(s)
I20260927 02:56:03.206727 17105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:56:03.207666 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.207341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:47225)
0927 02:56:03.207467 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.207533 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
W20260927 02:56:03.208029 17234 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45791: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
I20260927 02:56:03.208238 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.208163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:35087)
0927 02:56:03.208197 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.208221 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 02:56:03.208467 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:03.208404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:46861)
0927 02:56:03.208435 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:03.208454 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 02:56:03.208563 17234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:56:03.209146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 17105
I20260927 02:56:04.014060 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:04.013811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:43193)
0927 02:56:04.013957 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:04.013992 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260927 02:56:04.084679 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:04.084480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:36351)
0927 02:56:04.084603 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:04.084638 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260927 02:56:04.209021 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:04.208845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:45213)
0927 02:56:04.208942 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:04.208975 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260927 02:56:04.493116 17241 raft_consensus.cc:482] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:56:04.493233 17241 raft_consensus.cc:504] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:04.493582 17241 leader_election.cc:288] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173), 0f948ebb84c144a38c3de9116cce83ed (127.0.98.2:45319)
Sep 27 02:56:04 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 02:56:04 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 02:56:04.509248 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:04.493868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43091 (local address 127.0.98.2:45319)
0927 02:56:04.494023 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:04.494031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:04.494046 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:56:04.494141 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:04.494143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:04.494180 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:04.494291 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:04.495202 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:04.496575 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:04.498059 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:04.498691 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:04.498822 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:04.505608 (+ 6786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:04.505650 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:04.505650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:04.505662 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:04.507497 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:04.507966 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:04.507969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:04.507969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:04.508076 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:04.508333 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:04.508333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:04.508334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:04.508620 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:04.508697 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:04.509022 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 02:56:04.509149 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":53,"threads_started":1}
I20260927 02:56:04.509259 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:04.493870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43291 (local address 127.0.98.3:39173)
0927 02:56:04.493993 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:04.494005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:04.494041 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:56:04.494538 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:04.494542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:04.494590 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:04.494725 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:04.495182 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:04.496569 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:04.498060 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:04.498691 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:04.498848 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:04.505919 (+ 7071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:04.505924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:04.505924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:04.505937 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:04.507721 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:04.508047 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:04.508050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:04.508050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:04.508146 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:04.508324 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:04.508325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:04.508325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:04.508620 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:04.508697 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:04.509022 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 02:56:04.509148 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
I20260927 02:56:04.509256 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:04.493724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39173 (local address 127.0.98.1:43291)
0927 02:56:04.493796 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:04.493809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:04.493868 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:04.494710 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:04.494711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:04.494915 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:04.495070 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:04.495077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:04.496679 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:04.496680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:04.497976 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:04.497979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:04.498812 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:04.498813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:04.498948 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:04.503808 (+ 4860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:04.503818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:04.505819 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:04.507791 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:04.507792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:04.507792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:04.507980 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:04.508200 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:04.508201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:04.508201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:04.508272 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:04.508689 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:04.508695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:04.508854 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 02:56:04.508966 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 02:56:04.509135 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":83}
I20260927 02:56:04.509315 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:04.493727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.98.1:43091)
0927 02:56:04.493842 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:04.493853 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:04.493923 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:04.494357 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:04.494359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:04.494836 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:04.495071 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:04.495078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:04.496681 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:04.496682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:04.497976 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:04.497979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:04.498804 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:04.498805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:04.498944 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:04.503808 (+ 4864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:04.503818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:04.505487 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:04.507573 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:04.507580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:04.507581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:04.507890 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:04.508138 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:04.508138 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:04.508139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:04.508263 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:04.508690 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:04.508695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:04.508854 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 02:56:04.508966 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 02:56:04.509199 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":185,"thread_start_us":49,"threads_started":1}
I20260927 02:56:04.509900 17054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561d9341ac464772b6d7aef937194a52" candidate_uuid: "dd6001bd73ec4f61963644600e60619d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0f948ebb84c144a38c3de9116cce83ed" is_pre_election: true
I20260927 02:56:04.510051 17054 raft_consensus.cc:2454] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd6001bd73ec4f61963644600e60619d in term 1.
W20260927 02:56:04.510113 16853 leader_election.cc:341] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173): Not found: Tablet not found: 561d9341ac464772b6d7aef937194a52
I20260927 02:56:04.510298 16852 leader_election.cc:302] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f948ebb84c144a38c3de9116cce83ed, dd6001bd73ec4f61963644600e60619d; no voters: 6af445c338d341fa85f6c4c8180b0aed
I20260927 02:56:04.510418 17241 raft_consensus.cc:2781] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:56:04.510478 17241 raft_consensus.cc:482] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:56:04.510500 17241 raft_consensus.cc:3037] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:56:04.511592 17241 raft_consensus.cc:504] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:04.511716 17241 leader_election.cc:288] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [CANDIDATE]: Term 2 election: Requested vote from peers 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173), 0f948ebb84c144a38c3de9116cce83ed (127.0.98.2:45319)
I20260927 02:56:04.511869 17054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561d9341ac464772b6d7aef937194a52" candidate_uuid: "dd6001bd73ec4f61963644600e60619d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0f948ebb84c144a38c3de9116cce83ed"
I20260927 02:56:04.511942 17054 raft_consensus.cc:3037] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 1 FOLLOWER]: Advancing to term 2
W20260927 02:56:04.511996 16853 leader_election.cc:341] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173): Not found: Tablet not found: 561d9341ac464772b6d7aef937194a52
I20260927 02:56:04.512986 17054 raft_consensus.cc:2454] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd6001bd73ec4f61963644600e60619d in term 2.
I20260927 02:56:04.513141 16852 leader_election.cc:302] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f948ebb84c144a38c3de9116cce83ed, dd6001bd73ec4f61963644600e60619d; no voters: 6af445c338d341fa85f6c4c8180b0aed
I20260927 02:56:04.513262 17241 raft_consensus.cc:2781] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:56:04.513389 17241 raft_consensus.cc:686] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [term 2 LEADER]: Becoming Leader. State: Replica: dd6001bd73ec4f61963644600e60619d, State: Running, Role: LEADER
I20260927 02:56:04.513470 17241 consensus_queue.cc:237] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:04.513654 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:04.513546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:50037)
0927 02:56:04.513584 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:04.513610 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:56:04.928473 16853 consensus_peers.cc:487] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d -> Peer 6af445c338d341fa85f6c4c8180b0aed (127.0.98.3:39173): Couldn't send request to peer 6af445c338d341fa85f6c4c8180b0aed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 561d9341ac464772b6d7aef937194a52. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:56:05.084825 17054 raft_consensus.cc:1261] T 561d9341ac464772b6d7aef937194a52 P 0f948ebb84c144a38c3de9116cce83ed [term 2 FOLLOWER]: Refusing update from remote peer dd6001bd73ec4f61963644600e60619d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 02:56:05.085256 17241 consensus_queue.cc:1035] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 02:56:05.085362 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.085106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:59443)
0927 02:56:05.085257 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:05.085323 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260927 02:56:05.086026 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.085950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:48707)
0927 02:56:05.085978 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:05.086004 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260927 02:56:05.086949 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.086873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:43979)
0927 02:56:05.086902 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:05.086925 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 02:56:05.087020 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.086848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:49287)
0927 02:56:05.086938 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:05.086969 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":49,"threads_started":1}
I20260927 02:56:05.209684 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.209437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:51611)
0927 02:56:05.209601 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:05.209642 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260927 02:56:05.506919 17255 ts_tablet_manager.cc:927] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Initiating tablet copy from peer dd6001bd73ec4f61963644600e60619d (127.0.98.1:46257)
I20260927 02:56:05.507160 17255 tablet_copy_client.cc:316] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:46257
Sep 27 02:56:05 dist-test-slave-mfx9 krb5kdc[15665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:56:05.513948 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.507523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59209 (local address 127.0.98.1:46257)
0927 02:56:05.507662 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:05.507666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:05.507675 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:56:05.507697 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:05.507699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:05.507711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:05.507783 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:05.508194 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:05.508873 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:05.509594 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:05.509881 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:05.509928 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:05.511742 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:05.511749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:05.511750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:05.511755 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:05.513010 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:05.513334 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:05.513336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:05.513336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:05.513385 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:05.513516 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:05.513516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:05.513517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:05.513654 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:05.513710 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:05.513827 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:56:05.513883 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260927 02:56:05.513950 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.507395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46257 (local address 127.0.98.3:59209)
0927 02:56:05.507440 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:05.507456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:05.507526 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:05.507785 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:05.507788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:05.508049 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:05.508107 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:05.508109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:05.508950 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:05.508951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:05.509533 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:05.509535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:05.509931 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:05.509932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:05.509980 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:05.510397 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:05.510401 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:05.511661 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:05.513087 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:05.513093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:05.513094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:05.513273 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:05.513430 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:05.513430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:05.513431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:05.513478 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:05.513696 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:05.513700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:05.513745 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:56:05.513799 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:56:05.513877 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:56:05.514354 16939 tablet_copy_service.cc:140] P dd6001bd73ec4f61963644600e60619d: Received BeginTabletCopySession request for tablet 561d9341ac464772b6d7aef937194a52 from peer 6af445c338d341fa85f6c4c8180b0aed ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59209)
I20260927 02:56:05.514415 16939 tablet_copy_service.cc:161] P dd6001bd73ec4f61963644600e60619d: Beginning new tablet copy session on tablet 561d9341ac464772b6d7aef937194a52 from peer 6af445c338d341fa85f6c4c8180b0aed at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59209: session id = 6af445c338d341fa85f6c4c8180b0aed-561d9341ac464772b6d7aef937194a52
I20260927 02:56:05.514900 16939 tablet_copy_source_session.cc:205] T 561d9341ac464772b6d7aef937194a52 P dd6001bd73ec4f61963644600e60619d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:56:05.515772 17255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561d9341ac464772b6d7aef937194a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:05.518246 17255 tablet_copy_client.cc:791] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: tablet copy: Starting download of 0 data blocks...
I20260927 02:56:05.518365 17255 tablet_copy_client.cc:655] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: tablet copy: Starting download of 1 WAL segments...
I20260927 02:56:05.519107 17255 tablet_copy_client.cc:531] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:56:05.519719 17255 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Bootstrap starting.
I20260927 02:56:05.520918 17255 log.cc:824] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:05.523072 17255 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:56:05.523272 17255 tablet_bootstrap.cc:492] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Bootstrap complete.
I20260927 02:56:05.523411 17255 ts_tablet_manager.cc:1397] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:56:05.524263 17255 raft_consensus.cc:348] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:05.524408 17255 raft_consensus.cc:729] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af445c338d341fa85f6c4c8180b0aed, State: Initialized, Role: FOLLOWER
I20260927 02:56:05.524523 17255 consensus_queue.cc:260] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af445c338d341fa85f6c4c8180b0aed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39173 } } peers { permanent_uuid: "0f948ebb84c144a38c3de9116cce83ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "dd6001bd73ec4f61963644600e60619d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46257 } }
I20260927 02:56:05.524739 17255 ts_tablet_manager.cc:1428] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:56:05.524993 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.524906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:50889)
0927 02:56:05.524938 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:05.524967 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 02:56:05.524986 16939 tablet_copy_service.cc:342] P dd6001bd73ec4f61963644600e60619d: Request end of tablet copy session 6af445c338d341fa85f6c4c8180b0aed-561d9341ac464772b6d7aef937194a52 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59209
I20260927 02:56:05.525030 16939 tablet_copy_service.cc:433] P dd6001bd73ec4f61963644600e60619d: ending tablet copy session 6af445c338d341fa85f6c4c8180b0aed-561d9341ac464772b6d7aef937194a52 on tablet 561d9341ac464772b6d7aef937194a52 with peer 6af445c338d341fa85f6c4c8180b0aed
I20260927 02:56:05.525273 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:05.525179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:50947)
0927 02:56:05.525221 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:05.525245 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:56:06.087483 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:06.087296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:36633)
0927 02:56:06.087411 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:06.087440 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
I20260927 02:56:06.087517 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:06.087352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:42387)
0927 02:56:06.087451 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:06.087478 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
I20260927 02:56:06.102237 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:06.102117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:35525)
0927 02:56:06.102165 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:06.102190 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:56:06.102273 17188 raft_consensus.cc:1203] T 561d9341ac464772b6d7aef937194a52 P 6af445c338d341fa85f6c4c8180b0aed [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 02:56:06.102813 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:06.102627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:43211)
0927 02:56:06.102709 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:06.102745 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":44,"threads_started":1}
I20260927 02:56:07.088148 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:07.087905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:58563)
0927 02:56:07.088069 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:07.088106 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260927 02:56:07.103070 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:07.102875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:38141)
0927 02:56:07.102996 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:07.103024 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260927 02:56:07.103353 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:07.103182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:44963)
0927 02:56:07.103287 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:07.103318 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
I20260927 02:56:08.088739 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:08.088553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:53297)
0927 02:56:08.088664 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:08.088697 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260927 02:56:08.103672 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:08.103479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:36243)
0927 02:56:08.103595 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:08.103628 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260927 02:56:08.103890 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:08.103708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:59115)
0927 02:56:08.103823 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:08.103857 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260927 02:56:08.103992 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:08.103904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:58437)
0927 02:56:08.103947 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:08.103971 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:56:09.089322 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:09.089121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:59717)
0927 02:56:09.089229 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:09.089262 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
I20260927 02:56:09.104489 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:09.104281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:44367)
0927 02:56:09.104397 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:09.104434 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260927 02:56:09.104534 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:09.104362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:47309)
0927 02:56:09.104462 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:09.104494 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":46,"threads_started":1}
I20260927 02:56:10.089884 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:10.089686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:36493)
0927 02:56:10.089810 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:10.089844 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260927 02:56:10.105095 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:10.104888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:49641)
0927 02:56:10.105026 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:10.105057 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260927 02:56:10.105086 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:10.104889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:43963)
0927 02:56:10.105002 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:10.105035 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260927 02:56:11.090495 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:11.090284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:37625)
0927 02:56:11.090416 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:11.090452 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260927 02:56:11.105756 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:11.105542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:52541)
0927 02:56:11.105681 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:11.105713 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260927 02:56:11.105803 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:11.105542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:36075)
0927 02:56:11.105727 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:11.105760 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260927 02:56:12.091074 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:12.090881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:41553)
0927 02:56:12.090998 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:12.091031 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260927 02:56:12.106395 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:12.106199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:34691)
0927 02:56:12.106317 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:12.106351 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260927 02:56:12.106403 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:12.106199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:33579)
0927 02:56:12.106340 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:12.106367 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260927 02:56:13.091629 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:13.091442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:47161)
0927 02:56:13.091554 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:13.091586 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260927 02:56:13.107038 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:13.106821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:59185)
0927 02:56:13.106947 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:13.106988 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260927 02:56:13.107038 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:13.106844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:33385)
0927 02:56:13.106951 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:13.106988 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":45,"threads_started":1}
I20260927 02:56:14.092278 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:14.092057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:38759)
0927 02:56:14.092192 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:14.092233 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260927 02:56:14.107663 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:14.107456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:33619)
0927 02:56:14.107573 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:14.107607 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260927 02:56:14.107673 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:14.107455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:58003)
0927 02:56:14.107603 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:14.107638 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260927 02:56:15.092875 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:15.092667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:45239)
0927 02:56:15.092793 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:15.092833 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260927 02:56:15.108263 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:15.108070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:35907)
0927 02:56:15.108193 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:15.108221 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260927 02:56:15.108269 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:15.108090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:60753)
0927 02:56:15.108201 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:15.108231 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260927 02:56:16.093532 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:16.093317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:44963)
0927 02:56:16.093451 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:16.093488 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260927 02:56:16.108856 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:16.108642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:40231)
0927 02:56:16.108766 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:16.108812 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260927 02:56:16.108894 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:16.108693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:39573)
0927 02:56:16.108806 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:16.108838 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260927 02:56:17.094106 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:17.093913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:50847)
0927 02:56:17.094030 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:17.094062 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260927 02:56:17.109468 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:17.109278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:44509)
0927 02:56:17.109391 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:17.109422 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260927 02:56:17.109473 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:17.109257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:38837)
0927 02:56:17.109400 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:17.109433 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260927 02:56:18.094734 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:18.094507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:45979)
0927 02:56:18.094651 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:18.094689 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260927 02:56:18.110133 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:18.109914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:50781)
0927 02:56:18.110067 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:18.110097 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260927 02:56:18.110132 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:18.109925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:51523)
0927 02:56:18.110045 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:18.110079 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260927 02:56:18.216564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 16801
I20260927 02:56:18.220904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 16968
I20260927 02:56:18.224749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 17105
2026-09-27T02:56:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (57999 ms)
[----------] 30 tests from SecurityITest (224813 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:18 dist-test-slave-mfx9 krb5kdc[17304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:18 dist-test-slave-mfx9 krb5kdc[17304](info): set up 2 sockets
Sep 27 02:56:18 dist-test-slave-mfx9 krb5kdc[17304](info): commencing operation
krb5kdc: starting...
W20260927 02:56:19.186143 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:19.291620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:19.368422 17317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:19.368603 17317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:19.368631 17317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:19.369714 17317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:19.369752 17317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:19.369771 17317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:19.369794 17317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:19.369810 17317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:19.371274 17317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:19.371500 17317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:19.371685 17317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:19.376197 17322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:19.376232 17317 server_base.cc:1034] running on GCE node
W20260927 02:56:19.376267 17323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:19.376420 17325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:19.376626 17317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:19.376881 17317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:19.378021 17317 hybrid_clock.cc:648] HybridClock initialized: now 1790477779378008 us; error 30 us; skew 500 ppm
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:19.380204 17317 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:19.380592 17317 webserver.cc:460] Webserver started at http://127.0.98.62:39973/ using document root <none> and password file <none>
I20260927 02:56:19.380774 17317 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:19.380822 17317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:19.380937 17317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:19.381887 17317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "a854b3aed74542f6b7e626315bab16d6"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "4350d7d1db9b077581a94569122006c7"
server_key_iv: "d7a6f6620b11fc73212936ba378fd010"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.382181 17317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "a854b3aed74542f6b7e626315bab16d6"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "4350d7d1db9b077581a94569122006c7"
server_key_iv: "d7a6f6620b11fc73212936ba378fd010"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.383356 17317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:19.383930 17332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.384073 17317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:19.384148 17317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "a854b3aed74542f6b7e626315bab16d6"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "4350d7d1db9b077581a94569122006c7"
server_key_iv: "d7a6f6620b11fc73212936ba378fd010"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.384202 17317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:19.407120 17317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:19.408627 17317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:19.408746 17317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:19.412250 17317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37287
I20260927 02:56:19.412292 17384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37287 every 8 connection(s)
I20260927 02:56:19.412563 17317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:19.413307 17385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:19.415162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 17317
I20260927 02:56:19.415258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:19.415436 392 external_mini_cluster.cc:1428] Setting key 697afdfbf1b12d5fab836f43380a2ced
I20260927 02:56:19.415624 17385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6: Bootstrap starting.
I20260927 02:56:19.416209 17385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:19.416469 17385 log.cc:824] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:19.417165 17385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6: No bootstrap required, opened a new log
I20260927 02:56:19.419140 17385 raft_consensus.cc:348] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a854b3aed74542f6b7e626315bab16d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260927 02:56:19.419301 17385 raft_consensus.cc:374] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:19.419373 17385 raft_consensus.cc:729] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a854b3aed74542f6b7e626315bab16d6, State: Initialized, Role: FOLLOWER
I20260927 02:56:19.419497 17385 consensus_queue.cc:260] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a854b3aed74542f6b7e626315bab16d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260927 02:56:19.419576 17385 raft_consensus.cc:388] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:19.419600 17385 raft_consensus.cc:482] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:19.419643 17385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:19.420210 17385 raft_consensus.cc:504] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a854b3aed74542f6b7e626315bab16d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260927 02:56:19.420303 17385 leader_election.cc:302] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a854b3aed74542f6b7e626315bab16d6; no voters:
I20260927 02:56:19.420460 17385 leader_election.cc:288] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:19.420514 17390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:19.420657 17390 raft_consensus.cc:686] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [term 1 LEADER]: Becoming Leader. State: Replica: a854b3aed74542f6b7e626315bab16d6, State: Running, Role: LEADER
I20260927 02:56:19.420702 17385 sys_catalog.cc:564] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:19.420773 17390 consensus_queue.cc:237] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a854b3aed74542f6b7e626315bab16d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260927 02:56:19.421213 17391 sys_catalog.cc:455] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a854b3aed74542f6b7e626315bab16d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a854b3aed74542f6b7e626315bab16d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260927 02:56:19.421288 17391 sys_catalog.cc:458] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:19.421494 17392 sys_catalog.cc:455] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a854b3aed74542f6b7e626315bab16d6. Latest consensus state: current_term: 1 leader_uuid: "a854b3aed74542f6b7e626315bab16d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a854b3aed74542f6b7e626315bab16d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260927 02:56:19.421553 17392 sys_catalog.cc:458] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:19.421872 17397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:19.422317 17397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:19.423730 17397 catalog_manager.cc:1347] Generated new cluster ID: b8dabad399d24f7394273d070f2b5c32
I20260927 02:56:19.423784 17397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:19.425701 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.415879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.62:37287)
0927 02:56:19.416043 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.416052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.416083 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:56:19.416511 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.416520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.416579 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.416717 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:19.417432 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.418274 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.419338 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.419880 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.419961 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.422671 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:19.422675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:19.422675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:19.422685 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:19.424336 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.424647 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.424650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.424651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.424752 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.424927 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.424927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.424928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.425141 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:19.425272 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.425389 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.425512 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260927 02:56:19.444626 17397 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:19.445302 17397 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:19.449371 17397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6: Generated new TSK 0
I20260927 02:56:19.449522 17397 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:56:19.477578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:19.554556 17411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:19.554740 17411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:19.554788 17411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:19.555912 17411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:19.555953 17411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:19.555977 17411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:56:19.557402 17411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:19.557613 17411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:19.557821 17411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:19.562462 17419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:19.562463 17417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:19.562466 17416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:19.562603 17411 server_base.cc:1034] running on GCE node
I20260927 02:56:19.562844 17411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:19.563057 17411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:19.564184 17411 hybrid_clock.cc:648] HybridClock initialized: now 1790477779564170 us; error 32 us; skew 500 ppm
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:19.566464 17411 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:56:19.566939 17411 webserver.cc:460] Webserver started at http://127.0.98.1:44059/ using document root <none> and password file <none>
I20260927 02:56:19.567150 17411 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:19.567196 17411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:19.567317 17411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:19.568260 17411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "37026429e31d423080a1ff658591e0f4"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "36a8d6535cfbc63f0e3d1939070f1980"
server_key_iv: "d36d90aafcd211525fdd0bb4cced96a1"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.568567 17411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "37026429e31d423080a1ff658591e0f4"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "36a8d6535cfbc63f0e3d1939070f1980"
server_key_iv: "d36d90aafcd211525fdd0bb4cced96a1"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.569705 17411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:19.570314 17426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.570474 17411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:19.570549 17411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "37026429e31d423080a1ff658591e0f4"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "36a8d6535cfbc63f0e3d1939070f1980"
server_key_iv: "d36d90aafcd211525fdd0bb4cced96a1"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.570612 17411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:19.582973 17411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:19.584470 17411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:19.584589 17411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:19.584759 17411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:19.585021 17411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:19.585050 17411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.585076 17411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:19.585100 17411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.590780 17411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42375
I20260927 02:56:19.590823 17539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42375 every 8 connection(s)
I20260927 02:56:19.591099 17411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:56:19.591882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 17411
I20260927 02:56:19.591988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:56:19.592176 392 external_mini_cluster.cc:1428] Setting key 1c82fc7976d1ec15241733132d2533aa
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:56:19.600728 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.591678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48831 (local address 127.0.98.62:37287)
0927 02:56:19.591741 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.591744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.591998 (+ 254us) server_negotiation.cc:407] Connection header received
0927 02:56:19.593072 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.593072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.593081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.593128 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:19.593859 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.594337 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.595749 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.595941 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.595988 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.598096 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:19.598098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:19.598098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:19.598106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:19.599529 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.599878 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.599880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.599880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.599935 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.600123 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.600124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.600124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.600247 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:19.600318 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.600585 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.600655 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:56:19.600822 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.591647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.1:48831)
0927 02:56:19.591835 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:19.591881 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:19.592982 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:19.593214 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:19.593218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:19.593535 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:19.593782 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.593788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.594416 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.594417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.595655 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.595658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.596007 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.596009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.596145 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.596573 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:19.596582 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:19.598011 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:19.599603 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:19.599609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:19.599610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:19.599809 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:19.599993 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:19.599993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:19.599994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:19.600075 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:19.600318 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:19.600324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:19.600462 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 02:56:19.600543 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:56:19.600655 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260927 02:56:19.601251 17540 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260927 02:56:19.601338 17540 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:19.601492 17540 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260927 02:56:19.601924 17349 ts_manager.cc:194] Registered new tserver with Master: 37026429e31d423080a1ff658591e0f4 (127.0.98.1:42375)
I20260927 02:56:19.602566 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:56:19.616958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:19.693122 17545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:19.693311 17545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:19.693331 17545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:19.694391 17545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:19.694422 17545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:19.694444 17545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:56:19.695822 17545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:19.695983 17545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:19.696139 17545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:19.700614 17551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:19.700625 17550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:19.700685 17553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:19.700733 17545 server_base.cc:1034] running on GCE node
I20260927 02:56:19.701081 17545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:19.701274 17545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:19.702404 17545 hybrid_clock.cc:648] HybridClock initialized: now 1790477779702390 us; error 29 us; skew 500 ppm
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:19.704739 17545 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:56:19.705104 17545 webserver.cc:460] Webserver started at http://127.0.98.2:35589/ using document root <none> and password file <none>
I20260927 02:56:19.705265 17545 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:19.705302 17545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:19.705390 17545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:19.706328 17545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "381aeb6a664e454d993cc850b4405f24"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "968d2f1bbb06629d75daf1f8c152959b"
server_key_iv: "db55f8d3debd1c853fcbbfdee9e6dd3d"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.706614 17545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "381aeb6a664e454d993cc850b4405f24"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "968d2f1bbb06629d75daf1f8c152959b"
server_key_iv: "db55f8d3debd1c853fcbbfdee9e6dd3d"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.707760 17545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 02:56:19.708357 17560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.708485 17545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:19.708563 17545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "381aeb6a664e454d993cc850b4405f24"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "968d2f1bbb06629d75daf1f8c152959b"
server_key_iv: "db55f8d3debd1c853fcbbfdee9e6dd3d"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.708621 17545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:19.733085 17545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:19.735319 17545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:19.735462 17545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:19.735682 17545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:19.735976 17545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:19.736011 17545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.736039 17545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:19.736056 17545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.741192 17545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46803
I20260927 02:56:19.741254 17673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46803 every 8 connection(s)
I20260927 02:56:19.741530 17545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:19.750010 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.742197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36421 (local address 127.0.98.62:37287)
0927 02:56:19.742255 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.742257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.742436 (+ 179us) server_negotiation.cc:407] Connection header received
0927 02:56:19.743406 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.743407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.743419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.743466 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:19.744077 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.744574 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.745542 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.745741 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.745796 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.747736 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:19.747738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:19.747738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:19.747746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:19.748823 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.749160 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.749162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.749162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.749214 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.749427 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.749428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.749428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.749531 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:19.749586 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.749836 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.749920 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:56:19.750141 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.742161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.2:36421)
0927 02:56:19.742317 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:19.742352 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:19.743297 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:19.743545 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:19.743549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:19.743797 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:19.743984 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.743989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.744654 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.744655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.745456 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.745458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.745808 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.745809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.745918 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.746320 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:19.746327 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:19.747650 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:19.748894 (+ 1244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:19.748900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:19.748903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:19.749096 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:19.749265 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:19.749266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:19.749266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:19.749372 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:19.749586 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:19.749592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:19.749718 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:56:19.749801 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:56:19.749942 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260927 02:56:19.750545 17674 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260927 02:56:19.750625 17674 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:19.750789 17674 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260927 02:56:19.751132 17349 ts_manager.cc:194] Registered new tserver with Master: 381aeb6a664e454d993cc850b4405f24 (127.0.98.2:46803)
I20260927 02:56:19.751148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 17545
I20260927 02:56:19.751230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:56:19.751401 392 external_mini_cluster.cc:1428] Setting key bca70531912c48b75ff0dbd2eb78bfb1
I20260927 02:56:19.751513 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36421
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:56:19.773468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:19.850934 17679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:19.851114 17679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:19.851135 17679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:19.852217 17679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:19.852248 17679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:19.852270 17679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:56:19.853653 17679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:19.853823 17679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:19.853986 17679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:56:19.858567 17679 server_base.cc:1034] running on GCE node
W20260927 02:56:19.858508 17687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:19.858508 17685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:19.858512 17684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:19.858946 17679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:19.859166 17679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:19.860322 17679 hybrid_clock.cc:648] HybridClock initialized: now 1790477779860306 us; error 27 us; skew 500 ppm
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:19.862519 17679 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:56:19.862952 17679 webserver.cc:460] Webserver started at http://127.0.98.3:40827/ using document root <none> and password file <none>
I20260927 02:56:19.863139 17679 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:19.863193 17679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:19.863317 17679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:19.864251 17679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "538c61189bd64fe89b0b0f3f1a214c8f"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "d21f2b52498f820d2fa06f16d5acec82"
server_key_iv: "157411e590ab3ad5288b8ed923389396"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.864552 17679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "538c61189bd64fe89b0b0f3f1a214c8f"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "d21f2b52498f820d2fa06f16d5acec82"
server_key_iv: "157411e590ab3ad5288b8ed923389396"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.865679 17679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:19.866276 17694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.866416 17679 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:19.866492 17679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "538c61189bd64fe89b0b0f3f1a214c8f"
format_stamp: "Formatted at 2026-09-27 02:56:19 on dist-test-slave-mfx9"
server_key: "d21f2b52498f820d2fa06f16d5acec82"
server_key_iv: "157411e590ab3ad5288b8ed923389396"
server_key_version: "encryptionkey@0"
I20260927 02:56:19.866554 17679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:19.886976 17679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:19.888562 17679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:19.888687 17679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:19.888880 17679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:19.889142 17679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:19.889189 17679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.889215 17679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:19.889240 17679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:19.894430 17679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42675
I20260927 02:56:19.894465 17807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42675 every 8 connection(s)
I20260927 02:56:19.894721 17679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:56:19.897574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 17679
I20260927 02:56:19.897684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:56:19.897877 392 external_mini_cluster.cc:1428] Setting key f835017863a5a827058a453cff86c6a8
Sep 27 02:56:19 dist-test-slave-mfx9 krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:19.903113 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.895314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44177 (local address 127.0.98.62:37287)
0927 02:56:19.895382 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.895384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.895521 (+ 137us) server_negotiation.cc:407] Connection header received
0927 02:56:19.896338 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.896339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.896349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.896396 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:19.896908 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.897467 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.898722 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.898970 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.899021 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.900900 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:19.900901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:19.900901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:19.900908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:19.902063 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.902405 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.902406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.902406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.902455 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.902640 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.902641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.902641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.902729 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:19.902804 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.902998 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.903053 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:56:19.903271 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.895282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.3:44177)
0927 02:56:19.895420 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:19.895452 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:19.896264 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:19.896447 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:19.896450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:19.896675 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:19.896841 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.896845 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.897543 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.897544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.898654 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.898657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.899037 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.899038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.899135 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.899553 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:19.899561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:19.900822 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:19.902134 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:19.902139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:19.902141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:19.902333 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:19.902504 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:19.902505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:19.902505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:19.902591 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:19.902795 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:19.902799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:19.902907 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 02:56:19.902983 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:56:19.903085 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260927 02:56:19.903698 17808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260927 02:56:19.903777 17808 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:19.903929 17808 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260927 02:56:19.904299 17349 ts_manager.cc:194] Registered new tserver with Master: 538c61189bd64fe89b0b0f3f1a214c8f (127.0.98.3:42675)
I20260927 02:56:19.904641 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44177
I20260927 02:56:19.909324 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:56:19.915678 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.910548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.0.98.62:37287)
0927 02:56:19.910596 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.910597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.910677 (+ 80us) server_negotiation.cc:407] Connection header received
0927 02:56:19.910788 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.910789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.910797 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.910837 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:19.911406 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.911852 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.912612 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.912792 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.912843 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.913790 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:19.913791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:19.913791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:19.913797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:19.914717 (+ 920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.915014 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.915015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.915016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.915060 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:19.915217 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:19.915218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:19.915218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:19.915343 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:19.915426 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.915531 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.915599 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:56:19.916930 17349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:56:19.918357 17349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:56:19.924274 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.920313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.0.98.3:42675)
0927 02:56:19.920456 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.920463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.920568 (+ 105us) server_negotiation.cc:407] Connection header received
0927 02:56:19.920646 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.920648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.920659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.920761 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:19.920936 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.922009 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.923150 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.923789 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.923834 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.923884 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.924140 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.924179 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260927 02:56:19.924316 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.920165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46803 (local address 127.0.0.1:33144)
0927 02:56:19.920475 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:19.920512 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:19.920574 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:19.920975 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:19.920975 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:19.920977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:19.921010 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.921011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.922335 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.922336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.923422 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.923424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.924066 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.924067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.924134 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.924138 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:56:19.924191 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:56:19.924247 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":49,"threads_started":1}
I20260927 02:56:19.924388 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.920171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42375 (local address 127.0.0.1:54160)
0927 02:56:19.920654 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:19.920661 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:19.920700 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:19.920926 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:19.920928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:19.920930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:19.920980 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.920982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.921995 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.921996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.923300 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.923302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.924175 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.924176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.924225 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.924228 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:19.924270 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:56:19.924303 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":33,"thread_start_us":67,"threads_started":1}
I20260927 02:56:19.924600 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.920341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.0.98.2:46803)
0927 02:56:19.920790 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.920794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.920803 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:56:19.920884 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.920888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.920905 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.920980 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:19.921058 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.922261 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.923491 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.923990 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.924052 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.924107 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.924499 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.924543 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":84,"threads_started":1}
I20260927 02:56:19.924602 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.920392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.0.98.1:42375)
0927 02:56:19.920539 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.920542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.920736 (+ 194us) server_negotiation.cc:407] Connection header received
0927 02:56:19.920779 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.920781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.920795 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.920861 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:19.921075 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.921898 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.923381 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.924082 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.924171 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.924228 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.924295 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.924337 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260927 02:56:19.924643 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.920165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42675 (local address 127.0.0.1:42224)
0927 02:56:19.920515 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:19.920523 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:19.920575 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:19.920783 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:19.920785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:19.920788 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:19.920846 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.920848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.922115 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.922116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.923064 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.923066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.923943 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.923944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.924024 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.924036 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 02:56:19.924539 (+ 503us) client_negotiation.cc:240] Negotiation successful
0927 02:56:19.924580 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":36,"spinlock_wait_cycles":768,"thread_start_us":62,"threads_started":1}
I20260927 02:56:19.925204 17608 tablet_service.cc:1467] Processing CreateTablet for tablet f212a14a1cb448ee9c358715ba99bbb3 (DEFAULT_TABLE table=test-table [id=7659be1ffd804f58bd83edd4dc3154e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:19.925284 17742 tablet_service.cc:1467] Processing CreateTablet for tablet f212a14a1cb448ee9c358715ba99bbb3 (DEFAULT_TABLE table=test-table [id=7659be1ffd804f58bd83edd4dc3154e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:19.925483 17608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f212a14a1cb448ee9c358715ba99bbb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:19.925587 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f212a14a1cb448ee9c358715ba99bbb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:19.927850 17823 tablet_bootstrap.cc:492] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: Bootstrap starting.
I20260927 02:56:19.927940 17822 tablet_bootstrap.cc:492] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Bootstrap starting.
I20260927 02:56:19.928195 17474 tablet_service.cc:1467] Processing CreateTablet for tablet f212a14a1cb448ee9c358715ba99bbb3 (DEFAULT_TABLE table=test-table [id=7659be1ffd804f58bd83edd4dc3154e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:19.928494 17823 tablet_bootstrap.cc:654] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:19.928500 17474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f212a14a1cb448ee9c358715ba99bbb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:19.928573 17822 tablet_bootstrap.cc:654] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:19.928917 17823 log.cc:824] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:19.929051 17822 log.cc:824] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:19.929471 17823 tablet_bootstrap.cc:492] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: No bootstrap required, opened a new log
I20260927 02:56:19.929512 17823 ts_tablet_manager.cc:1397] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:19.930706 17823 raft_consensus.cc:348] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.930840 17826 tablet_bootstrap.cc:492] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4: Bootstrap starting.
I20260927 02:56:19.930891 17823 raft_consensus.cc:374] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:19.930946 17823 raft_consensus.cc:729] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538c61189bd64fe89b0b0f3f1a214c8f, State: Initialized, Role: FOLLOWER
I20260927 02:56:19.931089 17823 consensus_queue.cc:260] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.931350 17823 ts_tablet_manager.cc:1428] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:19.931356 17808 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260927 02:56:19.931455 17826 tablet_bootstrap.cc:654] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:19.931990 17822 tablet_bootstrap.cc:492] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: No bootstrap required, opened a new log
I20260927 02:56:19.931986 17826 log.cc:824] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:19.932067 17822 ts_tablet_manager.cc:1397] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 02:56:19.932655 17826 tablet_bootstrap.cc:492] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4: No bootstrap required, opened a new log
I20260927 02:56:19.932708 17826 ts_tablet_manager.cc:1397] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:19.933516 17822 raft_consensus.cc:348] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.933637 17822 raft_consensus.cc:374] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:19.933656 17822 raft_consensus.cc:729] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381aeb6a664e454d993cc850b4405f24, State: Initialized, Role: FOLLOWER
I20260927 02:56:19.933765 17822 consensus_queue.cc:260] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.933993 17822 ts_tablet_manager.cc:1428] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:19.934013 17674 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260927 02:56:19.934340 17826 raft_consensus.cc:348] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.934476 17826 raft_consensus.cc:374] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:19.934520 17826 raft_consensus.cc:729] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37026429e31d423080a1ff658591e0f4, State: Initialized, Role: FOLLOWER
I20260927 02:56:19.934620 17826 consensus_queue.cc:260] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.934898 17540 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260927 02:56:19.934883 17826 ts_tablet_manager.cc:1428] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 02:56:19.989032 17830 raft_consensus.cc:482] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:56:19.989131 17830 raft_consensus.cc:504] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.989459 17830 leader_election.cc:288] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 538c61189bd64fe89b0b0f3f1a214c8f (127.0.98.3:42675), 381aeb6a664e454d993cc850b4405f24 (127.0.98.2:46803)
W20260927 02:56:19.992027 17675 tablet.cc:2369] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:56:19.992254 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.989589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42675 (local address 127.0.98.1:60309)
0927 02:56:19.989662 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:19.989673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:19.989715 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:19.989837 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:19.989838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:19.989840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:19.989911 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.989913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.990573 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.990574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.991377 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.991379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.992018 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.992019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.992096 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.992099 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:19.992146 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:56:19.992185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 02:56:19.992317 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.989653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59067 (local address 127.0.98.2:46803)
0927 02:56:19.989697 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.989699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.989894 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:56:19.989935 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.989936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.989948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.989997 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:19.990138 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.990591 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.991530 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.991994 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.992060 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.992068 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.992231 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.992265 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:56:19.992323 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.989655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60309 (local address 127.0.98.3:42675)
0927 02:56:19.989731 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:19.989733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:19.989740 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:19.989764 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:19.989764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:19.989771 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:19.989821 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:19.989972 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.990508 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.991456 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.991940 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.992004 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.992015 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:19.992219 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 02:56:19.992262 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:56:19.992465 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:19.989590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46803 (local address 127.0.98.1:59067)
0927 02:56:19.989845 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:19.989853 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:19.989895 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:19.990013 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:19.990014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:19.990015 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:19.990081 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.990082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.990659 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.990659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.991405 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:19.991407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:19.992096 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:19.992096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:19.992161 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:19.992163 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:56:19.992377 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 02:56:19.992415 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":59,"thread_start_us":133,"threads_started":1}
I20260927 02:56:19.992694 17762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f212a14a1cb448ee9c358715ba99bbb3" candidate_uuid: "37026429e31d423080a1ff658591e0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" is_pre_election: true
I20260927 02:56:19.992736 17628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f212a14a1cb448ee9c358715ba99bbb3" candidate_uuid: "37026429e31d423080a1ff658591e0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381aeb6a664e454d993cc850b4405f24" is_pre_election: true
I20260927 02:56:19.992837 17628 raft_consensus.cc:2454] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37026429e31d423080a1ff658591e0f4 in term 0.
I20260927 02:56:19.992846 17762 raft_consensus.cc:2454] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37026429e31d423080a1ff658591e0f4 in term 0.
I20260927 02:56:19.993045 17427 leader_election.cc:302] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37026429e31d423080a1ff658591e0f4, 381aeb6a664e454d993cc850b4405f24; no voters:
I20260927 02:56:19.993196 17830 raft_consensus.cc:2781] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:56:19.993250 17830 raft_consensus.cc:482] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:56:19.993268 17830 raft_consensus.cc:3037] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:19.993908 17830 raft_consensus.cc:504] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.994019 17830 leader_election.cc:288] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [CANDIDATE]: Term 1 election: Requested vote from peers 538c61189bd64fe89b0b0f3f1a214c8f (127.0.98.3:42675), 381aeb6a664e454d993cc850b4405f24 (127.0.98.2:46803)
I20260927 02:56:19.994171 17762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f212a14a1cb448ee9c358715ba99bbb3" candidate_uuid: "37026429e31d423080a1ff658591e0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538c61189bd64fe89b0b0f3f1a214c8f"
I20260927 02:56:19.994180 17628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f212a14a1cb448ee9c358715ba99bbb3" candidate_uuid: "37026429e31d423080a1ff658591e0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381aeb6a664e454d993cc850b4405f24"
I20260927 02:56:19.994240 17762 raft_consensus.cc:3037] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:19.994247 17628 raft_consensus.cc:3037] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:19.994943 17628 raft_consensus.cc:2454] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37026429e31d423080a1ff658591e0f4 in term 1.
I20260927 02:56:19.994964 17762 raft_consensus.cc:2454] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37026429e31d423080a1ff658591e0f4 in term 1.
I20260927 02:56:19.995088 17427 leader_election.cc:302] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37026429e31d423080a1ff658591e0f4, 381aeb6a664e454d993cc850b4405f24, 538c61189bd64fe89b0b0f3f1a214c8f; no voters:
I20260927 02:56:19.995218 17830 raft_consensus.cc:2781] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:19.995359 17830 raft_consensus.cc:686] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 37026429e31d423080a1ff658591e0f4, State: Running, Role: LEADER
I20260927 02:56:19.995440 17830 consensus_queue.cc:237] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } }
I20260927 02:56:19.996162 17349 catalog_manager.cc:5495] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37026429e31d423080a1ff658591e0f4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "37026429e31d423080a1ff658591e0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37026429e31d423080a1ff658591e0f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42375 } health_report { overall_health: HEALTHY } } }
I20260927 02:56:20.013938 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.011867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.0.98.1:42375)
0927 02:56:20.011915 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.011917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.011926 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:56:20.011993 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.011994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.012010 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.012083 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:20.012183 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.012660 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.013339 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.013520 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.013561 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.013660 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:20.013754 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:20.013810 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.013849 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.013866 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:56:20.015575 17628 raft_consensus.cc:1261] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 1 FOLLOWER]: Refusing update from remote peer 37026429e31d423080a1ff658591e0f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:20.015851 17762 raft_consensus.cc:1261] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 1 FOLLOWER]: Refusing update from remote peer 37026429e31d423080a1ff658591e0f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:20.015897 17830 consensus_queue.cc:1035] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381aeb6a664e454d993cc850b4405f24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:20.016080 17832 consensus_queue.cc:1035] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "538c61189bd64fe89b0b0f3f1a214c8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:20.018169 17835 mvcc.cc:205] Tried to move back new op lower bound from 7333796986941304832 to 7333796986861826048. Current Snapshot: MvccSnapshot[applied={T|T < 7333796986941304832}]
I20260927 02:56:20.018291 17836 mvcc.cc:205] Tried to move back new op lower bound from 7333796986941304832 to 7333796986861826048. Current Snapshot: MvccSnapshot[applied={T|T < 7333796986941304832}]
I20260927 02:56:20.018327 17837 mvcc.cc:205] Tried to move back new op lower bound from 7333796986941304832 to 7333796986861826048. Current Snapshot: MvccSnapshot[applied={T|T < 7333796986941304832}]
I20260927 02:56:20.020617 17349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:20.020694 17349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:20.021322 17349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a854b3aed74542f6b7e626315bab16d6: Sending DeleteTablet for 3 replicas of tablet f212a14a1cb448ee9c358715ba99bbb3
I20260927 02:56:20.021651 17608 tablet_service.cc:1514] Processing DeleteTablet for tablet f212a14a1cb448ee9c358715ba99bbb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33144
I20260927 02:56:20.021660 17742 tablet_service.cc:1514] Processing DeleteTablet for tablet f212a14a1cb448ee9c358715ba99bbb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42224
I20260927 02:56:20.021853 17846 tablet_replica.cc:330] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: stopping tablet replica
I20260927 02:56:20.021884 17474 tablet_service.cc:1514] Processing DeleteTablet for tablet f212a14a1cb448ee9c358715ba99bbb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54160
I20260927 02:56:20.021965 17846 raft_consensus.cc:2229] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:20.022065 17846 raft_consensus.cc:2258] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:20.022182 17847 tablet_replica.cc:330] T f212a14a1cb448ee9c358715ba99bbb3 P 37026429e31d423080a1ff658591e0f4: stopping tablet replica
I20260927 02:56:20.022281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 17411
I20260927 02:56:20.022300 17845 tablet_replica.cc:330] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: stopping tablet replica
I20260927 02:56:20.022378 17845 raft_consensus.cc:2229] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:20.022469 17845 raft_consensus.cc:2258] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:20.023185 17846 ts_tablet_manager.cc:1905] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:20.024420 17845 ts_tablet_manager.cc:1905] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:20.024677 17846 ts_tablet_manager.cc:1918] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:20.024823 17846 log.cc:1196] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/f212a14a1cb448ee9c358715ba99bbb3
I20260927 02:56:20.025086 17846 ts_tablet_manager.cc:1939] T f212a14a1cb448ee9c358715ba99bbb3 P 381aeb6a664e454d993cc850b4405f24: Deleting consensus metadata
I20260927 02:56:20.025539 17333 catalog_manager.cc:4841] TS 381aeb6a664e454d993cc850b4405f24 (127.0.98.2:46803): tablet f212a14a1cb448ee9c358715ba99bbb3 (table test-table [id=7659be1ffd804f58bd83edd4dc3154e6]) successfully deleted
I20260927 02:56:20.026007 17845 ts_tablet_manager.cc:1918] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:20.026064 17845 log.cc:1196] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/f212a14a1cb448ee9c358715ba99bbb3
I20260927 02:56:20.026296 17845 ts_tablet_manager.cc:1939] T f212a14a1cb448ee9c358715ba99bbb3 P 538c61189bd64fe89b0b0f3f1a214c8f: Deleting consensus metadata
I20260927 02:56:20.026640 17334 catalog_manager.cc:4841] TS 538c61189bd64fe89b0b0f3f1a214c8f (127.0.98.3:42675): tablet f212a14a1cb448ee9c358715ba99bbb3 (table test-table [id=7659be1ffd804f58bd83edd4dc3154e6]) successfully deleted
W20260927 02:56:20.027669 17336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42375 (error 108)
I20260927 02:56:20.027849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 17545
I20260927 02:56:20.028106 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.028015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42375 (local address 127.0.0.1:54186)
0927 02:56:20.028054 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.028078 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:56:20.028235 17336 catalog_manager.cc:4584] TS 37026429e31d423080a1ff658591e0f4 (127.0.98.1:42375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f212a14a1cb448ee9c358715ba99bbb3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42375: connect: Connection refused (error 111)
I20260927 02:56:20.032356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 17679
I20260927 02:56:20.035892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 17317
2026-09-27T02:56:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1810 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T02:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:20Z Disabled control of system clock
I20260927 02:56:20.060257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32789
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:20.136637 17852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:20.136837 17852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:20.136863 17852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:20.137925 17852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:20.137955 17852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:20.137966 17852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:20.137977 17852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:20.137987 17852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:20.139267 17852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:20.139488 17852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:20.139720 17852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:20.144129 17860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:20.144163 17857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:20.144207 17858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:20.144351 17852 server_base.cc:1034] running on GCE node
I20260927 02:56:20.144608 17852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:20.144869 17852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:20.146008 17852 hybrid_clock.cc:648] HybridClock initialized: now 1790477780145995 us; error 28 us; skew 500 ppm
I20260927 02:56:20.147142 17852 webserver.cc:460] Webserver started at http://127.0.98.62:38719/ using document root <none> and password file <none>
I20260927 02:56:20.147425 17852 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:20.147473 17852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:20.147593 17852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:20.148603 17852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "ea20631b01db4a509bbf9fcd063e4d27"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "ffbac4944bc842f5faae350283cd1eaa"
server_key_iv: "4f1660ff9c08e295cab687403edcea4e"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.148914 17852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea20631b01db4a509bbf9fcd063e4d27"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "ffbac4944bc842f5faae350283cd1eaa"
server_key_iv: "4f1660ff9c08e295cab687403edcea4e"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.150027 17852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:20.150626 17866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.150800 17852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:20.150884 17852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "ea20631b01db4a509bbf9fcd063e4d27"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "ffbac4944bc842f5faae350283cd1eaa"
server_key_iv: "4f1660ff9c08e295cab687403edcea4e"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.150950 17852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:20.163887 17852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:20.164121 17852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:20.164230 17852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:20.167663 17852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32789
I20260927 02:56:20.167690 17918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32789 every 8 connection(s)
I20260927 02:56:20.167958 17852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:20.168797 17919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:20.170841 17919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27: Bootstrap starting.
I20260927 02:56:20.171331 17919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:20.171553 17919 log.cc:824] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:20.172101 17919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27: No bootstrap required, opened a new log
I20260927 02:56:20.173537 17919 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea20631b01db4a509bbf9fcd063e4d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260927 02:56:20.173681 17919 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:20.173702 17919 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea20631b01db4a509bbf9fcd063e4d27, State: Initialized, Role: FOLLOWER
I20260927 02:56:20.173804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 17852
I20260927 02:56:20.173832 17919 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea20631b01db4a509bbf9fcd063e4d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260927 02:56:20.173899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:20.173920 17919 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:20.173959 17919 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:20.174005 17919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:20.174067 392 external_mini_cluster.cc:1428] Setting key d590eebe61e268dfd0841f28a9e73480
I20260927 02:56:20.174530 17919 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea20631b01db4a509bbf9fcd063e4d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260927 02:56:20.174641 17919 leader_election.cc:302] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea20631b01db4a509bbf9fcd063e4d27; no voters:
I20260927 02:56:20.174808 17919 leader_election.cc:288] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:20.174834 17924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:20.174973 17924 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [term 1 LEADER]: Becoming Leader. State: Replica: ea20631b01db4a509bbf9fcd063e4d27, State: Running, Role: LEADER
I20260927 02:56:20.175058 17919 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:20.175073 17924 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea20631b01db4a509bbf9fcd063e4d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260927 02:56:20.175500 17925 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea20631b01db4a509bbf9fcd063e4d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea20631b01db4a509bbf9fcd063e4d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } } }
I20260927 02:56:20.175589 17925 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:20.175783 17926 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea20631b01db4a509bbf9fcd063e4d27. Latest consensus state: current_term: 1 leader_uuid: "ea20631b01db4a509bbf9fcd063e4d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea20631b01db4a509bbf9fcd063e4d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } } }
I20260927 02:56:20.176146 17926 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:20.176271 17933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:20.176745 17933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:20.178944 17933 catalog_manager.cc:1347] Generated new cluster ID: 2e83cc0121854c73b543e48ce2ce3f48
I20260927 02:56:20.178996 17933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:20.179633 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.174504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.0.98.62:32789)
0927 02:56:20.174783 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.174792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.174822 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:56:20.174933 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.174941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.175004 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.175165 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:20.175383 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.176809 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.177625 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.178186 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.178305 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.178951 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:20.178955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:20.178955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:20.179140 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:20.179180 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:20.179193 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:20.179252 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.179358 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.179428 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260927 02:56:20.189235 17933 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:20.189764 17933 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:20.194092 17933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27: Generated new TSK 0
I20260927 02:56:20.194228 17933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:20.197405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:20.273497 17943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:20.273679 17943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:20.273698 17943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:20.274857 17943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:20.274894 17943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:20.274917 17943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:56:20.276286 17943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:20.276497 17943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:20.276769 17943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:20.281203 17951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:20.281217 17948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:20.281445 17943 server_base.cc:1034] running on GCE node
W20260927 02:56:20.281493 17949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:20.281713 17943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:20.281921 17943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:20.283068 17943 hybrid_clock.cc:648] HybridClock initialized: now 1790477780283057 us; error 25 us; skew 500 ppm
I20260927 02:56:20.284224 17943 webserver.cc:460] Webserver started at http://127.0.98.1:32979/ using document root <none> and password file <none>
I20260927 02:56:20.284440 17943 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:20.284482 17943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:20.284593 17943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:20.285558 17943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "6e3de8d512ce461ea6e0755996f5f94a"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "b789874eacf74352e2db0dc4590f4e6f"
server_key_iv: "f9ec6aa047111143ef01292a339d7d28"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.285862 17943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6e3de8d512ce461ea6e0755996f5f94a"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "b789874eacf74352e2db0dc4590f4e6f"
server_key_iv: "f9ec6aa047111143ef01292a339d7d28"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.287050 17943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:20.287719 17957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.287869 17943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:20.287961 17943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "6e3de8d512ce461ea6e0755996f5f94a"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "b789874eacf74352e2db0dc4590f4e6f"
server_key_iv: "f9ec6aa047111143ef01292a339d7d28"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.288023 17943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:20.318168 17943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:20.318437 17943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:20.318535 17943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:20.318713 17943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:20.319025 17943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:20.319057 17943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.319083 17943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:20.319099 17943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.324301 17943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33893
I20260927 02:56:20.324371 18070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33893 every 8 connection(s)
I20260927 02:56:20.324656 17943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:56:20.329465 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.325264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36339 (local address 127.0.98.62:32789)
0927 02:56:20.325322 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.325325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.325524 (+ 199us) server_negotiation.cc:407] Connection header received
0927 02:56:20.326027 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.326028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.326040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.326090 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:20.326742 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.327257 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.328196 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.328401 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.328454 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.329196 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:20.329198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:20.329198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:20.329203 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:20.329219 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:20.329223 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:20.329267 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.329353 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.329387 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:56:20.329643 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.325246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.1:36339)
0927 02:56:20.325409 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.325441 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:20.325928 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:20.326158 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:20.326162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:20.326172 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:20.326653 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.326657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.327337 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.327338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.328112 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.328114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.328470 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.328471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.328596 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.329067 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:20.329068 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:20.329095 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:20.329116 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:20.329281 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:20.329286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:20.329289 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:20.329349 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:56:20.329447 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260927 02:56:20.330065 18071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32789
I20260927 02:56:20.330140 18071 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:20.330279 18071 heartbeater.cc:511] Master 127.0.98.62:32789 requested a full tablet report, sending...
I20260927 02:56:20.330691 17883 ts_manager.cc:194] Registered new tserver with Master: 6e3de8d512ce461ea6e0755996f5f94a (127.0.98.1:33893)
I20260927 02:56:20.331267 17883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36339
I20260927 02:56:20.331816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 17943
I20260927 02:56:20.331899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:56:20.332052 392 external_mini_cluster.cc:1428] Setting key 9da3ad6486dd6978c8f127ee73256445
I20260927 02:56:20.332787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:20.411293 18074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:20.411516 18074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:20.411536 18074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:20.412650 18074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:20.412688 18074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:20.412709 18074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:56:20.414000 18074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:20.414204 18074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:20.414589 18074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:20.419224 18079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:20.419255 18080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:20.419232 18082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:20.419436 18074 server_base.cc:1034] running on GCE node
I20260927 02:56:20.419608 18074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:20.419780 18074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:20.420899 18074 hybrid_clock.cc:648] HybridClock initialized: now 1790477780420890 us; error 25 us; skew 500 ppm
I20260927 02:56:20.421977 18074 webserver.cc:460] Webserver started at http://127.0.98.2:42295/ using document root <none> and password file <none>
I20260927 02:56:20.422168 18074 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:20.422210 18074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:20.422323 18074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:20.423326 18074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "6583360a7d064a00b38f9029f01fb988"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "8840c0f3452f3096d5b5853b5bb14adf"
server_key_iv: "79f5a97b18449a3d8d8f03fd6ff5c666"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.423625 18074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6583360a7d064a00b38f9029f01fb988"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "8840c0f3452f3096d5b5853b5bb14adf"
server_key_iv: "79f5a97b18449a3d8d8f03fd6ff5c666"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.424728 18074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:20.425324 18088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.425450 18074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:20.425526 18074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "6583360a7d064a00b38f9029f01fb988"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "8840c0f3452f3096d5b5853b5bb14adf"
server_key_iv: "79f5a97b18449a3d8d8f03fd6ff5c666"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.425581 18074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:20.439368 18074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:20.439623 18074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:20.439742 18074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:20.439946 18074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:20.440241 18074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:20.440274 18074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.440302 18074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:20.440318 18074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.445467 18074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44921
I20260927 02:56:20.445508 18201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44921 every 8 connection(s)
I20260927 02:56:20.445786 18074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:56:20.446918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 18074
I20260927 02:56:20.447029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:56:20.447207 392 external_mini_cluster.cc:1428] Setting key a26aead96f051abcff9faf11719b60f5
I20260927 02:56:20.448132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:56:20.451022 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.446432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60945 (local address 127.0.98.62:32789)
0927 02:56:20.446484 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.446487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.446777 (+ 290us) server_negotiation.cc:407] Connection header received
0927 02:56:20.447337 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.447339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.447354 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.447421 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:20.448155 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.448821 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.449727 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.449930 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.449985 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.450749 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:20.450750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:20.450750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:20.450777 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:20.450794 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:20.450798 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:20.450852 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.450919 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.450948 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:56:20.451162 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.446407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.2:60945)
0927 02:56:20.446582 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.446617 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:20.447241 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:20.447486 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:20.447490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:20.447499 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:20.448071 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.448077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.448899 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.448900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.449644 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.449646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.449995 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.449996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.450117 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.450614 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:20.450614 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:20.450640 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:20.450660 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:20.450853 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:20.450856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:20.450858 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:56:20.450912 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:56:20.450987 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260927 02:56:20.451563 18202 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32789
I20260927 02:56:20.451640 18202 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:20.451800 18202 heartbeater.cc:511] Master 127.0.98.62:32789 requested a full tablet report, sending...
I20260927 02:56:20.452119 17883 ts_manager.cc:194] Registered new tserver with Master: 6583360a7d064a00b38f9029f01fb988 (127.0.98.2:44921)
I20260927 02:56:20.452456 17883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60945
W20260927 02:56:20.525410 18205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:20.525597 18205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:20.525624 18205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:20.526783 18205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:20.526821 18205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:20.526860 18205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:56:20.528223 18205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:20.528450 18205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:20.528705 18205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:20.533329 18211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:20.533396 18205 server_base.cc:1034] running on GCE node
W20260927 02:56:20.533351 18210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:20.533351 18213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:20.533717 18205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:20.533964 18205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:20.535100 18205 hybrid_clock.cc:648] HybridClock initialized: now 1790477780535091 us; error 23 us; skew 500 ppm
I20260927 02:56:20.536187 18205 webserver.cc:460] Webserver started at http://127.0.98.3:41203/ using document root <none> and password file <none>
I20260927 02:56:20.536387 18205 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:20.536433 18205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:20.536561 18205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:20.537541 18205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "d06a4c91306848deb75f4e993c5a322b"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "2676306397157d369d9fa051ccb08552"
server_key_iv: "4ebfe620fe39508eaa49a706f644d3f2"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.537882 18205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d06a4c91306848deb75f4e993c5a322b"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "2676306397157d369d9fa051ccb08552"
server_key_iv: "4ebfe620fe39508eaa49a706f644d3f2"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.539064 18205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:20.539671 18219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.539821 18205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:20.539897 18205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "d06a4c91306848deb75f4e993c5a322b"
format_stamp: "Formatted at 2026-09-27 02:56:20 on dist-test-slave-mfx9"
server_key: "2676306397157d369d9fa051ccb08552"
server_key_iv: "4ebfe620fe39508eaa49a706f644d3f2"
server_key_version: "encryptionkey@0"
I20260927 02:56:20.539953 18205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:20.552690 18205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:20.552953 18205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:20.553067 18205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:20.553261 18205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:20.553530 18205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:20.553567 18205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.553593 18205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:20.553617 18205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:20.558620 18205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39281
I20260927 02:56:20.558684 18332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39281 every 8 connection(s)
I20260927 02:56:20.558965 18205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:56:20.562908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 18205
I20260927 02:56:20.563007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:56:20.563177 392 external_mini_cluster.cc:1428] Setting key 0c5c1a49bd3f571cb7b58a7be69aaf78
I20260927 02:56:20.563740 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.559553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52079 (local address 127.0.98.62:32789)
0927 02:56:20.559594 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.559596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.559806 (+ 210us) server_negotiation.cc:407] Connection header received
0927 02:56:20.560229 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.560230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.560240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.560285 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:20.560885 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.561370 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.562363 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.562556 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.562617 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.563480 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:20.563481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:20.563482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:20.563486 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:20.563504 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:20.563508 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:20.563558 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.563632 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.563660 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:56:20.563968 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.559538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.3:52079)
0927 02:56:20.559685 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.559717 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:20.560156 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:20.560337 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:20.560340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:20.560348 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:20.560825 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.560828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.561461 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.561462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.562257 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.562260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.562631 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.562632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.562777 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.563339 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:20.563340 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:20.563378 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:20.563404 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:20.563565 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:20.563569 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:20.563571 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:56:20.563631 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:56:20.563763 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260927 02:56:20.564349 18333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32789
I20260927 02:56:20.564420 18333 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:20.564576 18333 heartbeater.cc:511] Master 127.0.98.62:32789 requested a full tablet report, sending...
I20260927 02:56:20.564868 17883 ts_manager.cc:194] Registered new tserver with Master: d06a4c91306848deb75f4e993c5a322b (127.0.98.3:39281)
I20260927 02:56:20.565214 17883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52079
I20260927 02:56:20.574928 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:56:20.578679 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.576279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.0.98.62:32789)
0927 02:56:20.576330 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.576333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.576474 (+ 141us) server_negotiation.cc:407] Connection header received
0927 02:56:20.576512 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.576513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.576520 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.576561 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:20.576676 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.577129 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.577799 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.577970 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.578013 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.578509 (+ 496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:20.578510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:20.578510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:20.578514 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:20.578526 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:20.578530 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:20.578561 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.578593 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.578613 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:56:20.579968 17883 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:56:20.581459 17883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:56:20.587311 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.583333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39281 (local address 127.0.0.1:34782)
0927 02:56:20.583459 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.583486 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:20.583550 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:20.584003 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:20.584004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:20.584005 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:20.584052 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.584054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.585353 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.585354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.586294 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.586296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.587103 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.587104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.587157 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.587167 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 02:56:20.587213 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:56:20.587251 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":9,"thread_start_us":83,"threads_started":1}
I20260927 02:56:20.587342 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.583495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.0.98.3:39281)
0927 02:56:20.583874 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.583878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.583886 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:20.583920 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.583922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.583934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.584000 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:20.584142 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.585248 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.586364 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.587029 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.587096 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.587183 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.587225 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.587259 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":77,"threads_started":1}
I20260927 02:56:20.587903 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.583526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33893 (local address 127.0.0.1:42538)
0927 02:56:20.584101 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.584111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:20.584156 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:20.584393 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:20.584394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:20.584395 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:20.584428 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.584429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.585225 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.585226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.586543 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.586546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.587687 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.587688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.587745 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.587749 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:56:20.587804 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:56:20.587841 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":48,"thread_start_us":63,"threads_started":1}
I20260927 02:56:20.588095 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.583715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.0.98.1:33893)
0927 02:56:20.583871 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.583875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.584208 (+ 333us) server_negotiation.cc:407] Connection header received
0927 02:56:20.584251 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.584253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.584268 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.584333 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:20.584490 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.585150 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.586633 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.587355 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.587420 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.587509 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.587831 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.587871 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260927 02:56:20.588104 18267 tablet_service.cc:1467] Processing CreateTablet for tablet f0d52d29054c4d8aae64c023e785071e (DEFAULT_TABLE table=test-table [id=5c237534fb27409ba5ac3c9a94b3e330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:20.588505 18267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d52d29054c4d8aae64c023e785071e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:20.588591 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.583333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44921 (local address 127.0.0.1:59388)
0927 02:56:20.583447 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.583487 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:20.583550 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:20.583976 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:20.583979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:20.583981 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:20.584052 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.584055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.585300 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.585301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.586543 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.586546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.588380 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.588381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.588439 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.588443 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:56:20.588489 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:56:20.588528 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":77,"thread_start_us":61,"threads_started":1}
I20260927 02:56:20.588632 18005 tablet_service.cc:1467] Processing CreateTablet for tablet f0d52d29054c4d8aae64c023e785071e (DEFAULT_TABLE table=test-table [id=5c237534fb27409ba5ac3c9a94b3e330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:20.588858 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.583663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.0.98.2:44921)
0927 02:56:20.583815 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.583820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.583832 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:56:20.583870 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.583872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.583888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.583973 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:20.584142 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.585227 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.587542 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.588191 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.588254 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.588334 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.588741 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.588782 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260927 02:56:20.588969 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d52d29054c4d8aae64c023e785071e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:20.589527 18136 tablet_service.cc:1467] Processing CreateTablet for tablet f0d52d29054c4d8aae64c023e785071e (DEFAULT_TABLE table=test-table [id=5c237534fb27409ba5ac3c9a94b3e330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:20.589855 18136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d52d29054c4d8aae64c023e785071e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:20.591616 18347 tablet_bootstrap.cc:492] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a: Bootstrap starting.
I20260927 02:56:20.592315 18347 tablet_bootstrap.cc:654] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:20.592568 18347 log.cc:824] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:20.592717 18349 tablet_bootstrap.cc:492] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Bootstrap starting.
I20260927 02:56:20.592819 18348 tablet_bootstrap.cc:492] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: Bootstrap starting.
I20260927 02:56:20.593320 18347 tablet_bootstrap.cc:492] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a: No bootstrap required, opened a new log
I20260927 02:56:20.593338 18349 tablet_bootstrap.cc:654] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:20.593361 18348 tablet_bootstrap.cc:654] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:20.593379 18347 ts_tablet_manager.cc:1397] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:20.593629 18349 log.cc:824] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:20.593636 18348 log.cc:824] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:20.594301 18348 tablet_bootstrap.cc:492] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: No bootstrap required, opened a new log
I20260927 02:56:20.594347 18349 tablet_bootstrap.cc:492] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: No bootstrap required, opened a new log
I20260927 02:56:20.594359 18348 ts_tablet_manager.cc:1397] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:20.594400 18349 ts_tablet_manager.cc:1397] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:20.595854 18347 raft_consensus.cc:348] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.596020 18347 raft_consensus.cc:374] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:20.596055 18347 raft_consensus.cc:729] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3de8d512ce461ea6e0755996f5f94a, State: Initialized, Role: FOLLOWER
I20260927 02:56:20.596102 18348 raft_consensus.cc:348] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.596186 18347 consensus_queue.cc:260] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.596241 18348 raft_consensus.cc:374] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:20.596280 18348 raft_consensus.cc:729] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06a4c91306848deb75f4e993c5a322b, State: Initialized, Role: FOLLOWER
I20260927 02:56:20.596379 18348 consensus_queue.cc:260] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.596427 18347 ts_tablet_manager.cc:1428] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:56:20.596549 18071 heartbeater.cc:503] Master 127.0.98.62:32789 was elected leader, sending a full tablet report...
I20260927 02:56:20.596628 18348 ts_tablet_manager.cc:1428] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:20.596748 18333 heartbeater.cc:503] Master 127.0.98.62:32789 was elected leader, sending a full tablet report...
I20260927 02:56:20.596730 18349 raft_consensus.cc:348] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.596876 18349 raft_consensus.cc:374] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:20.596913 18349 raft_consensus.cc:729] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6583360a7d064a00b38f9029f01fb988, State: Initialized, Role: FOLLOWER
I20260927 02:56:20.597031 18349 consensus_queue.cc:260] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.597256 18349 ts_tablet_manager.cc:1428] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:56:20.597364 18202 heartbeater.cc:503] Master 127.0.98.62:32789 was elected leader, sending a full tablet report...
W20260927 02:56:20.696296 18203 tablet.cc:2369] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:56:20.733810 18355 raft_consensus.cc:482] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:56:20.734009 18355 raft_consensus.cc:504] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.734375 18355 leader_election.cc:288] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e3de8d512ce461ea6e0755996f5f94a (127.0.98.1:33893), d06a4c91306848deb75f4e993c5a322b (127.0.98.3:39281)
I20260927 02:56:20.738067 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.734593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58749 (local address 127.0.98.3:39281)
0927 02:56:20.734647 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.734649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.734797 (+ 148us) server_negotiation.cc:407] Connection header received
0927 02:56:20.734827 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.734828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.734837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.734884 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:20.734999 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.735721 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.737164 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.737716 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.737785 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.737795 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.737959 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.737998 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:56:20.738067 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.734588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32875 (local address 127.0.98.1:33893)
0927 02:56:20.734630 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.734632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.734644 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:56:20.734708 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.734708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.734719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.734821 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:20.734977 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.735750 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.737155 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.737716 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.737824 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.737833 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.737958 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.737996 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:56:20.738082 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.734521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33893 (local address 127.0.98.2:32875)
0927 02:56:20.734575 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.734587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:20.734626 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:20.734831 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:20.734832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:20.734833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:20.734911 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.734914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.735814 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.735815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.737067 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.737071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.737839 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.737839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.737903 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.737905 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:56:20.737951 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:56:20.737998 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":46}
I20260927 02:56:20.738087 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.734521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39281 (local address 127.0.98.2:58749)
0927 02:56:20.734690 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.734698 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:20.734739 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:20.734892 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:20.734893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:20.734894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:20.734942 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.734943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.735791 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.735792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.737070 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:20.737071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.737801 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.737802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:20.737890 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.737895 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:56:20.737947 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:56:20.737995 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":48,"thread_start_us":89,"threads_started":1}
I20260927 02:56:20.738461 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d52d29054c4d8aae64c023e785071e" candidate_uuid: "6583360a7d064a00b38f9029f01fb988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3de8d512ce461ea6e0755996f5f94a" is_pre_election: true
I20260927 02:56:20.738622 18025 raft_consensus.cc:2454] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6583360a7d064a00b38f9029f01fb988 in term 0.
I20260927 02:56:20.738660 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d52d29054c4d8aae64c023e785071e" candidate_uuid: "6583360a7d064a00b38f9029f01fb988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06a4c91306848deb75f4e993c5a322b" is_pre_election: true
I20260927 02:56:20.738852 18287 raft_consensus.cc:2454] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6583360a7d064a00b38f9029f01fb988 in term 0.
I20260927 02:56:20.738858 18089 leader_election.cc:302] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6583360a7d064a00b38f9029f01fb988, 6e3de8d512ce461ea6e0755996f5f94a; no voters:
I20260927 02:56:20.738963 18355 raft_consensus.cc:2781] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:56:20.739008 18355 raft_consensus.cc:482] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:56:20.739029 18355 raft_consensus.cc:3037] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:20.739681 18355 raft_consensus.cc:504] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.739801 18355 leader_election.cc:288] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [CANDIDATE]: Term 1 election: Requested vote from peers 6e3de8d512ce461ea6e0755996f5f94a (127.0.98.1:33893), d06a4c91306848deb75f4e993c5a322b (127.0.98.3:39281)
I20260927 02:56:20.739948 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d52d29054c4d8aae64c023e785071e" candidate_uuid: "6583360a7d064a00b38f9029f01fb988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06a4c91306848deb75f4e993c5a322b"
I20260927 02:56:20.739945 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d52d29054c4d8aae64c023e785071e" candidate_uuid: "6583360a7d064a00b38f9029f01fb988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3de8d512ce461ea6e0755996f5f94a"
I20260927 02:56:20.740015 18287 raft_consensus.cc:3037] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:20.740015 18025 raft_consensus.cc:3037] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:20.740681 18025 raft_consensus.cc:2454] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6583360a7d064a00b38f9029f01fb988 in term 1.
I20260927 02:56:20.740732 18287 raft_consensus.cc:2454] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6583360a7d064a00b38f9029f01fb988 in term 1.
I20260927 02:56:20.740831 18089 leader_election.cc:302] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6583360a7d064a00b38f9029f01fb988, 6e3de8d512ce461ea6e0755996f5f94a; no voters:
I20260927 02:56:20.740933 18355 raft_consensus.cc:2781] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:20.741068 18355 raft_consensus.cc:686] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 1 LEADER]: Becoming Leader. State: Replica: 6583360a7d064a00b38f9029f01fb988, State: Running, Role: LEADER
I20260927 02:56:20.741161 18355 consensus_queue.cc:237] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } }
I20260927 02:56:20.741882 17881 catalog_manager.cc:5495] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6583360a7d064a00b38f9029f01fb988 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6583360a7d064a00b38f9029f01fb988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6583360a7d064a00b38f9029f01fb988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 } health_report { overall_health: UNKNOWN } } }
I20260927 02:56:20.767912 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.765979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.0.98.2:44921)
0927 02:56:20.766026 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:20.766028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:20.766035 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:20.766063 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:20.766064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:20.766079 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:20.766137 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:20.766223 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.766663 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.767345 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:20.767521 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:20.767566 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:20.767654 (+ 88us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:20.767742 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:20.767817 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:20.767829 (+ 12us) server_negotiation.cc:299] Negotiation successful
0927 02:56:20.767845 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:56:20.769431 18287 raft_consensus.cc:1261] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 1 FOLLOWER]: Refusing update from remote peer 6583360a7d064a00b38f9029f01fb988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:20.769461 18025 raft_consensus.cc:1261] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 1 FOLLOWER]: Refusing update from remote peer 6583360a7d064a00b38f9029f01fb988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:20.769727 18357 consensus_queue.cc:1035] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06a4c91306848deb75f4e993c5a322b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:20.769861 18355 consensus_queue.cc:1035] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3de8d512ce461ea6e0755996f5f94a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:20.771904 18360 mvcc.cc:205] Tried to move back new op lower bound from 7333796990029430784 to 7333796989916323840. Current Snapshot: MvccSnapshot[applied={T|T < 7333796990029430784}]
I20260927 02:56:20.772430 18361 mvcc.cc:205] Tried to move back new op lower bound from 7333796990029430784 to 7333796989916323840. Current Snapshot: MvccSnapshot[applied={T|T < 7333796990029430784 or (T in {7333796990029430784})}]
I20260927 02:56:20.774040 17881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:20.774127 17881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:20.774796 17881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea20631b01db4a509bbf9fcd063e4d27: Sending DeleteTablet for 3 replicas of tablet f0d52d29054c4d8aae64c023e785071e
I20260927 02:56:20.775139 18267 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d52d29054c4d8aae64c023e785071e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:20 UTC) from {username='slave'} at 127.0.0.1:34782
I20260927 02:56:20.775159 18005 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d52d29054c4d8aae64c023e785071e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:20 UTC) from {username='slave'} at 127.0.0.1:42538
I20260927 02:56:20.775153 18136 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d52d29054c4d8aae64c023e785071e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:20 UTC) from {username='slave'} at 127.0.0.1:59388
I20260927 02:56:20.775409 18372 tablet_replica.cc:330] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a: stopping tablet replica
I20260927 02:56:20.775506 18372 raft_consensus.cc:2229] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:20.775524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 17943
I20260927 02:56:20.775593 18372 raft_consensus.cc:2258] T f0d52d29054c4d8aae64c023e785071e P 6e3de8d512ce461ea6e0755996f5f94a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:20.775772 18370 tablet_replica.cc:330] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: stopping tablet replica
I20260927 02:56:20.775838 18370 raft_consensus.cc:2229] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:56:20.775899 18370 raft_consensus.cc:2258] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:20.776253 18370 ts_tablet_manager.cc:1905] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:20.777756 18370 ts_tablet_manager.cc:1918] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:20.777817 18370 log.cc:1196] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/f0d52d29054c4d8aae64c023e785071e
I20260927 02:56:20.778028 18370 ts_tablet_manager.cc:1939] T f0d52d29054c4d8aae64c023e785071e P 6583360a7d064a00b38f9029f01fb988: Deleting consensus metadata
I20260927 02:56:20.778513 17867 catalog_manager.cc:4841] TS 6583360a7d064a00b38f9029f01fb988 (127.0.98.2:44921): tablet f0d52d29054c4d8aae64c023e785071e (table test-table [id=5c237534fb27409ba5ac3c9a94b3e330]) successfully deleted
I20260927 02:56:20.778869 18371 tablet_replica.cc:330] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: stopping tablet replica
I20260927 02:56:20.778956 18371 raft_consensus.cc:2229] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:20.779048 18371 raft_consensus.cc:2258] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:20.779377 18371 ts_tablet_manager.cc:1905] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:20.780632 18371 ts_tablet_manager.cc:1918] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:20.780689 18371 log.cc:1196] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/f0d52d29054c4d8aae64c023e785071e
I20260927 02:56:20.780889 18371 ts_tablet_manager.cc:1939] T f0d52d29054c4d8aae64c023e785071e P d06a4c91306848deb75f4e993c5a322b: Deleting consensus metadata
I20260927 02:56:20.781216 17870 catalog_manager.cc:4841] TS d06a4c91306848deb75f4e993c5a322b (127.0.98.3:39281): tablet f0d52d29054c4d8aae64c023e785071e (table test-table [id=5c237534fb27409ba5ac3c9a94b3e330]) successfully deleted
W20260927 02:56:20.782068 17867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33893 (error 108)
I20260927 02:56:20.782249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 18074
I20260927 02:56:20.782748 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:20.782354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33893 (local address 127.0.0.1:42548)
0927 02:56:20.782687 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:20.782722 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316}
W20260927 02:56:20.782905 17867 catalog_manager.cc:4584] TS 6e3de8d512ce461ea6e0755996f5f94a (127.0.98.1:33893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0d52d29054c4d8aae64c023e785071e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33893: connect: Connection refused (error 111)
I20260927 02:56:20.786592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 18205
I20260927 02:56:20.790040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 17852
2026-09-27T02:56:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (753 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:20 dist-test-slave-mfx9 krb5kdc[18376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:20 dist-test-slave-mfx9 krb5kdc[18376](info): set up 2 sockets
Sep 27 02:56:20 dist-test-slave-mfx9 krb5kdc[18376](info): commencing operation
krb5kdc: starting...
W20260927 02:56:21.738245 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:21 dist-test-slave-mfx9 krb5kdc[18376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:21.845336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:21.923305 18389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:21.923494 18389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:21.923523 18389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:21.924624 18389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:21.924662 18389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:21.924680 18389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:21.924695 18389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:21.924711 18389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:21.926120 18389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:21.926347 18389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:21.926549 18389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:21.930997 18394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:21.930990 18397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:21.931011 18389 server_base.cc:1034] running on GCE node
W20260927 02:56:21.930994 18395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:21.931414 18389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:21.931689 18389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:21.932824 18389 hybrid_clock.cc:648] HybridClock initialized: now 1790477781932810 us; error 30 us; skew 500 ppm
Sep 27 02:56:21 dist-test-slave-mfx9 krb5kdc[18376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:21.935052 18389 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:21.935417 18389 webserver.cc:460] Webserver started at http://127.0.98.62:39443/ using document root <none> and password file <none>
I20260927 02:56:21.935611 18389 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:21.935655 18389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:21.935768 18389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:21.936717 18389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "0079349fdf9d4976b73720fd56339670"
format_stamp: "Formatted at 2026-09-27 02:56:21 on dist-test-slave-mfx9"
server_key: "ace6347414ad4ba8918f1b9ecf470b5a"
server_key_iv: "dc606f6d524bfd627f337e728ab1ff4c"
server_key_version: "encryptionkey@0"
I20260927 02:56:21.937037 18389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "0079349fdf9d4976b73720fd56339670"
format_stamp: "Formatted at 2026-09-27 02:56:21 on dist-test-slave-mfx9"
server_key: "ace6347414ad4ba8918f1b9ecf470b5a"
server_key_iv: "dc606f6d524bfd627f337e728ab1ff4c"
server_key_version: "encryptionkey@0"
I20260927 02:56:21.938170 18389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:56:21.938848 18404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:21.938993 18389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:21.939064 18389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "0079349fdf9d4976b73720fd56339670"
format_stamp: "Formatted at 2026-09-27 02:56:21 on dist-test-slave-mfx9"
server_key: "ace6347414ad4ba8918f1b9ecf470b5a"
server_key_iv: "dc606f6d524bfd627f337e728ab1ff4c"
server_key_version: "encryptionkey@0"
I20260927 02:56:21.939127 18389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:21.948138 18389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:21.949883 18389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:21.949994 18389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:21.953500 18389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34329
I20260927 02:56:21.953528 18456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34329 every 8 connection(s)
I20260927 02:56:21.953817 18389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:21.954584 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:21.956629 18457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670: Bootstrap starting.
I20260927 02:56:21.957125 18457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:21.957371 18457 log.cc:824] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:21.957950 18457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670: No bootstrap required, opened a new log
I20260927 02:56:21.959144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 18389
I20260927 02:56:21.959249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:21.959425 392 external_mini_cluster.cc:1428] Setting key 86cc1e5e3e876182bba531b4e56d2170
I20260927 02:56:21.959717 18457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0079349fdf9d4976b73720fd56339670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260927 02:56:21.959852 18457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:21.959885 18457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0079349fdf9d4976b73720fd56339670, State: Initialized, Role: FOLLOWER
I20260927 02:56:21.960000 18457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0079349fdf9d4976b73720fd56339670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260927 02:56:21.960081 18457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:21.960120 18457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:21.960163 18457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:21.960812 18457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0079349fdf9d4976b73720fd56339670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260927 02:56:21.960896 18457 leader_election.cc:302] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0079349fdf9d4976b73720fd56339670; no voters:
I20260927 02:56:21.961028 18457 leader_election.cc:288] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:21.961061 18462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:21.961164 18462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [term 1 LEADER]: Becoming Leader. State: Replica: 0079349fdf9d4976b73720fd56339670, State: Running, Role: LEADER
I20260927 02:56:21.961266 18457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:21.961241 18462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0079349fdf9d4976b73720fd56339670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260927 02:56:21.961690 18464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0079349fdf9d4976b73720fd56339670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0079349fdf9d4976b73720fd56339670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260927 02:56:21.961705 18463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0079349fdf9d4976b73720fd56339670. Latest consensus state: current_term: 1 leader_uuid: "0079349fdf9d4976b73720fd56339670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0079349fdf9d4976b73720fd56339670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260927 02:56:21.961875 18463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:21.962069 18471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:21.961846 18464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:21.962638 18471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:21.964383 18471 catalog_manager.cc:1347] Generated new cluster ID: 025be36b639440929d83ef3f3ef33e69
I20260927 02:56:21.964437 18471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:56:21 dist-test-slave-mfx9 krb5kdc[18376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:21.969889 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:21.959897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.0.98.62:34329)
0927 02:56:21.960072 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:21.960080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:21.960110 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:56:21.960218 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:21.960224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:21.960276 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:21.960430 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:21.961008 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:21.962333 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:21.963408 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:21.963865 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:21.963973 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:21.966546 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:21.966550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:21.966551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:21.966562 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:21.968384 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:21.968745 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:21.968748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:21.968748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:21.968861 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:21.969032 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:21.969033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:21.969033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:21.969332 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:21.969500 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:21.969567 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:56:21.969698 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260927 02:56:21.970800 18471 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:21.971482 18471 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:21.977723 18471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670: Generated new TSK 0
I20260927 02:56:21.977896 18471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:56:22.004791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:22.082345 18483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:22.082548 18483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:22.082574 18483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:22.083799 18483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:22.083843 18483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:22.083868 18483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:56:22.085269 18483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:22.085453 18483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:22.085619 18483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:22.090204 18488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.090152 18489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.090154 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:22.090273 18483 server_base.cc:1034] running on GCE node
I20260927 02:56:22.090576 18483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:22.090826 18483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:22.091985 18483 hybrid_clock.cc:648] HybridClock initialized: now 1790477782091972 us; error 28 us; skew 500 ppm
Sep 27 02:56:22 dist-test-slave-mfx9 krb5kdc[18376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:22.094251 18483 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:56:22.094637 18483 webserver.cc:460] Webserver started at http://127.0.98.1:33719/ using document root <none> and password file <none>
I20260927 02:56:22.094877 18483 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:22.094924 18483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:22.095041 18483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:22.095996 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "810fcd53a6b84dd895d63f15f7846eb7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "2aae674f56b0cd2fd80736a01dc3ae3b"
server_key_iv: "618fefd7dff3954d104c4646aa029fa1"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.096304 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "810fcd53a6b84dd895d63f15f7846eb7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "2aae674f56b0cd2fd80736a01dc3ae3b"
server_key_iv: "618fefd7dff3954d104c4646aa029fa1"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.097476 18483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:56:22.098073 18498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.098217 18483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.098294 18483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "810fcd53a6b84dd895d63f15f7846eb7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "2aae674f56b0cd2fd80736a01dc3ae3b"
server_key_iv: "618fefd7dff3954d104c4646aa029fa1"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.098361 18483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:22.113636 18483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:22.115384 18483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:22.115511 18483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:22.115702 18483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:22.115973 18483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:22.116005 18483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.116032 18483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:22.116050 18483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.121299 18483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36613
I20260927 02:56:22.121359 18611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36613 every 8 connection(s)
I20260927 02:56:22.121628 18483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:56:22 dist-test-slave-mfx9 krb5kdc[18376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:22.129101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 18483
I20260927 02:56:22.129215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:56:22.129419 392 external_mini_cluster.cc:1428] Setting key 00844d657c9ae705f22d1c8a37e98411
I20260927 02:56:22.130169 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.122271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35731 (local address 127.0.98.62:34329)
0927 02:56:22.122328 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.122330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.122514 (+ 184us) server_negotiation.cc:407] Connection header received
0927 02:56:22.123477 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.123478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.123490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.123542 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:22.124084 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.124577 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.125470 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.125648 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.125690 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.127755 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:22.127756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:22.127756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:22.127763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:22.128835 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:22.129246 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:22.129247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:22.129247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:22.129323 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:22.129577 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:22.129578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:22.129578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:22.129715 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:22.129772 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.129998 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.130093 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:56:22.130328 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.122250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.1:35731)
0927 02:56:22.122397 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.122438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.123394 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.123594 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.123597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.123831 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:22.124013 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.124018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.124637 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.124638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.125413 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.125415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.125700 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.125700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.125792 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.126248 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:22.126257 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:22.127691 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:22.128895 (+ 1204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:22.128901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:22.128902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:22.129177 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:22.129385 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:22.129386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:22.129387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:22.129518 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:22.129762 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:22.129767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:22.129878 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.129971 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.130114 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260927 02:56:22.130740 18612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260927 02:56:22.130851 18612 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:22.131014 18612 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260927 02:56:22.131546 18421 ts_manager.cc:194] Registered new tserver with Master: 810fcd53a6b84dd895d63f15f7846eb7 (127.0.98.1:36613)
I20260927 02:56:22.132365 18421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35731
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:56:22.151263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:22.228129 18617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:22.228338 18617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:22.228360 18617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:22.229496 18617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:22.229537 18617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:22.229565 18617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:56:22.231051 18617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:22.231261 18617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:22.231469 18617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:22.236130 18625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.236150 18622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.236166 18623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:22.236272 18617 server_base.cc:1034] running on GCE node
I20260927 02:56:22.236562 18617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:22.236783 18617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:22.237906 18617 hybrid_clock.cc:648] HybridClock initialized: now 1790477782237899 us; error 26 us; skew 500 ppm
Sep 27 02:56:22 dist-test-slave-mfx9 krb5kdc[18376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:22.240223 18617 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:56:22.240648 18617 webserver.cc:460] Webserver started at http://127.0.98.2:36683/ using document root <none> and password file <none>
I20260927 02:56:22.240840 18617 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:22.240881 18617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:22.240994 18617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:22.242043 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "6613dc6c96404c04a0c172ac3610e01f"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "0ab33f903f9effabacdea3ad790e7d12"
server_key_iv: "9ed7c9a11554a7c92dd9736b3f19ce5a"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.242353 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6613dc6c96404c04a0c172ac3610e01f"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "0ab33f903f9effabacdea3ad790e7d12"
server_key_iv: "9ed7c9a11554a7c92dd9736b3f19ce5a"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.243566 18617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 02:56:22.244194 18632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.244342 18617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.244422 18617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "6613dc6c96404c04a0c172ac3610e01f"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "0ab33f903f9effabacdea3ad790e7d12"
server_key_iv: "9ed7c9a11554a7c92dd9736b3f19ce5a"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.244474 18617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:22.279789 18617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:22.281937 18617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:22.282048 18617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:22.282236 18617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:22.282492 18617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:22.282521 18617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.282541 18617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:22.282557 18617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.287801 18617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36193
I20260927 02:56:22.287864 18745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36193 every 8 connection(s)
I20260927 02:56:22.288139 18617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:56:22 dist-test-slave-mfx9 krb5kdc[18376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:22.295392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 18617
I20260927 02:56:22.295506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:56:22.295692 392 external_mini_cluster.cc:1428] Setting key 209915ba15b4d58186f4898753245738
I20260927 02:56:22.296993 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.288771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56795 (local address 127.0.98.62:34329)
0927 02:56:22.288823 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.288825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.288994 (+ 169us) server_negotiation.cc:407] Connection header received
0927 02:56:22.289884 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.289886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.289899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.289948 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:22.290549 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.291105 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.292070 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.292301 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.292367 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.294255 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:22.294256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:22.294256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:22.294263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:22.295477 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:22.295889 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:22.295890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:22.295890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:22.295950 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:22.296389 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:22.296390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:22.296390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:22.296521 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:22.296589 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.296821 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.296904 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:56:22.297061 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.288735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.2:56795)
0927 02:56:22.288881 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.288917 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.289773 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.290019 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.290023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.290261 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:22.290456 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.290460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.291184 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.291185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.291983 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.291985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.292373 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.292375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.292500 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.292928 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:22.292936 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:22.294166 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:22.295554 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:22.295561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:22.295563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:22.295818 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:22.296193 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:22.296193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:22.296194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:22.296327 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:22.296585 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:22.296590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:22.296701 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.296791 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.296905 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260927 02:56:22.297483 18746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260927 02:56:22.297566 18746 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:22.297731 18746 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260927 02:56:22.298105 18421 ts_manager.cc:194] Registered new tserver with Master: 6613dc6c96404c04a0c172ac3610e01f (127.0.98.2:36193)
I20260927 02:56:22.298452 18421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56795
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:56:22.317296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:22.394104 18751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:22.394294 18751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:22.394315 18751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:22.395452 18751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:22.395491 18751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:22.395516 18751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:56:22.396936 18751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:22.397147 18751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:22.397363 18751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:22.401957 18756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.401947 18759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.401950 18757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:22.402205 18751 server_base.cc:1034] running on GCE node
I20260927 02:56:22.402376 18751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:22.402603 18751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:22.403734 18751 hybrid_clock.cc:648] HybridClock initialized: now 1790477782403709 us; error 41 us; skew 500 ppm
Sep 27 02:56:22 dist-test-slave-mfx9 krb5kdc[18376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:22.405915 18751 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:56:22.406319 18751 webserver.cc:460] Webserver started at http://127.0.98.3:34891/ using document root <none> and password file <none>
I20260927 02:56:22.406515 18751 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:22.406558 18751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:22.406672 18751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:22.407689 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "c80286bad3de95ccbc29b045bfa283fd"
server_key_iv: "7ec258862d6ba8e3732a80046c9cf9df"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.407992 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "c80286bad3de95ccbc29b045bfa283fd"
server_key_iv: "7ec258862d6ba8e3732a80046c9cf9df"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.409189 18751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:56:22.409904 18766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.410056 18751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:22.410132 18751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "c80286bad3de95ccbc29b045bfa283fd"
server_key_iv: "7ec258862d6ba8e3732a80046c9cf9df"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.410192 18751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:22.451395 18751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:22.453362 18751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:22.453495 18751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:22.453707 18751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:22.454010 18751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:22.454042 18751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.454072 18751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:22.454089 18751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.459510 18751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37403
I20260927 02:56:22.459568 18879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37403 every 8 connection(s)
I20260927 02:56:22.459846 18751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:56:22.461620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 18751
I20260927 02:56:22.461727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:56:22.461936 392 external_mini_cluster.cc:1428] Setting key e228ac90f9f4bfe696039a6f9588a9d7
Sep 27 02:56:22 dist-test-slave-mfx9 krb5kdc[18376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:22.468510 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.460514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34817 (local address 127.0.98.62:34329)
0927 02:56:22.460556 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.460558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.460744 (+ 186us) server_negotiation.cc:407] Connection header received
0927 02:56:22.461796 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.461797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.461810 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.461865 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:22.462646 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.463224 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.464173 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.464360 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.464405 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.466339 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:22.466340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:22.466340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:22.466347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:22.467555 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:22.467853 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:22.467854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:22.467854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:22.467898 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:22.468068 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:22.468068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:22.468068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:22.468156 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:22.468197 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.468398 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.468449 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:22.468674 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.460493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.3:34817)
0927 02:56:22.460644 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.460678 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.461622 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.461969 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.461975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.462290 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:22.462573 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.462579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.463302 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.463303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.464101 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.464103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.464415 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.464415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.464514 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.464995 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:22.465003 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:22.466265 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:22.467609 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:22.467613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:22.467614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:22.467802 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:22.467939 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:22.467939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:22.467940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:22.468027 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:22.468195 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:22.468198 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:22.468300 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.468380 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.468490 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260927 02:56:22.469110 18880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260927 02:56:22.469187 18880 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:22.469370 18880 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260927 02:56:22.469694 18421 ts_manager.cc:194] Registered new tserver with Master: 1fd90a0645db4448bb9d7fe07b4c3bd7 (127.0.98.3:37403)
I20260927 02:56:22.470062 18421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34817
I20260927 02:56:22.473461 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:56:22.479694 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.474586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.0.98.62:34329)
0927 02:56:22.474642 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.474644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.474728 (+ 84us) server_negotiation.cc:407] Connection header received
0927 02:56:22.474820 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.474821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.474830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.474877 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:22.475327 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.475789 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.476449 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.476624 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.476671 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.477688 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:22.477689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:22.477689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:22.477695 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:22.478864 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:22.479180 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:22.479181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:22.479181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:22.479224 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:22.479341 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:22.479341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:22.479342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:22.479445 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:22.479514 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.479581 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.479623 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:56:22.481119 18421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:56:22.482563 18421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:56:22.488224 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.484569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37403 (local address 127.0.0.1:42996)
0927 02:56:22.484702 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.484739 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.484802 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.485145 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.485149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.485152 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:22.485211 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.485213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.486130 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.486131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.487080 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.487082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.487936 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.487937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.488022 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.488037 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.488106 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.488151 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260927 02:56:22.488242 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.484864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.0.98.3:37403)
0927 02:56:22.485005 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.485009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.485018 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:56:22.485050 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.485052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.485063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.485130 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:22.485266 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.486042 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.487163 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.487863 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.487939 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.487998 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.488123 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.488163 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260927 02:56:22.488859 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.484738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36193 (local address 127.0.0.1:59090)
0927 02:56:22.485071 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.485080 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.485131 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.485359 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.485360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.485362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:22.485409 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.485411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.486745 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.486746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.487743 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.487745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.488636 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.488637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.488695 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.488698 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.488749 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.488786 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":26,"thread_start_us":61,"threads_started":1}
I20260927 02:56:22.488868 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.484932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.0.98.2:36193)
0927 02:56:22.485072 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.485077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.485121 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:56:22.485189 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.485191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.485207 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.485276 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:22.485558 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.486637 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.487813 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.488562 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.488624 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.488676 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.488762 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.488798 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260927 02:56:22.488958 18814 tablet_service.cc:1467] Processing CreateTablet for tablet a8470efbe9cf42c39fc7aaeee224c52f (DEFAULT_TABLE table=test-table [id=c585381c5f5e416aac0a2b244bf12376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:22.489260 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.484557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36613 (local address 127.0.0.1:56156)
0927 02:56:22.484700 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.484739 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.484804 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.485421 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.485422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.485423 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:22.485456 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.485457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.486792 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.486793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.487887 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.487889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.489071 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.489072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.489131 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.489134 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.489167 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.489200 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":18,"thread_start_us":57,"threads_started":1}
I20260927 02:56:22.489316 18814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8470efbe9cf42c39fc7aaeee224c52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:22.489610 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.484697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.98.1:36613)
0927 02:56:22.485243 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.485246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.485255 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:56:22.485319 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.485320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.485335 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.485521 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:22.485595 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.486666 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.487971 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.488855 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.489413 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.489476 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.489501 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.489540 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":80,"threads_started":1}
I20260927 02:56:22.489578 18680 tablet_service.cc:1467] Processing CreateTablet for tablet a8470efbe9cf42c39fc7aaeee224c52f (DEFAULT_TABLE table=test-table [id=c585381c5f5e416aac0a2b244bf12376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:22.489909 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8470efbe9cf42c39fc7aaeee224c52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:22.491191 18546 tablet_service.cc:1467] Processing CreateTablet for tablet a8470efbe9cf42c39fc7aaeee224c52f (DEFAULT_TABLE table=test-table [id=c585381c5f5e416aac0a2b244bf12376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:22.491498 18546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8470efbe9cf42c39fc7aaeee224c52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:22.491838 18897 tablet_bootstrap.cc:492] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: Bootstrap starting.
I20260927 02:56:22.492408 18897 tablet_bootstrap.cc:654] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:22.492756 18898 tablet_bootstrap.cc:492] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Bootstrap starting.
I20260927 02:56:22.492933 18897 log.cc:824] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:22.493319 18898 tablet_bootstrap.cc:654] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:22.493705 18897 tablet_bootstrap.cc:492] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: No bootstrap required, opened a new log
I20260927 02:56:22.493760 18897 ts_tablet_manager.cc:1397] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:22.493820 18898 log.cc:824] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:22.494534 18898 tablet_bootstrap.cc:492] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: No bootstrap required, opened a new log
I20260927 02:56:22.494601 18898 ts_tablet_manager.cc:1397] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:22.494186 18901 tablet_bootstrap.cc:492] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7: Bootstrap starting.
I20260927 02:56:22.495278 18901 tablet_bootstrap.cc:654] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:22.495721 18901 log.cc:824] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:22.495842 18897 raft_consensus.cc:348] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.496021 18897 raft_consensus.cc:374] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:22.496069 18897 raft_consensus.cc:729] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd90a0645db4448bb9d7fe07b4c3bd7, State: Initialized, Role: FOLLOWER
I20260927 02:56:22.496201 18897 consensus_queue.cc:260] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.496387 18901 tablet_bootstrap.cc:492] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7: No bootstrap required, opened a new log
I20260927 02:56:22.496446 18901 ts_tablet_manager.cc:1397] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:22.496462 18897 ts_tablet_manager.cc:1428] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:56:22.496582 18880 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260927 02:56:22.496754 18898 raft_consensus.cc:348] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.496917 18898 raft_consensus.cc:374] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:22.496955 18898 raft_consensus.cc:729] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6613dc6c96404c04a0c172ac3610e01f, State: Initialized, Role: FOLLOWER
I20260927 02:56:22.497057 18898 consensus_queue.cc:260] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.497323 18898 ts_tablet_manager.cc:1428] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:56:22.497462 18746 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260927 02:56:22.498135 18901 raft_consensus.cc:348] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.498301 18901 raft_consensus.cc:374] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:22.498342 18901 raft_consensus.cc:729] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810fcd53a6b84dd895d63f15f7846eb7, State: Initialized, Role: FOLLOWER
I20260927 02:56:22.498435 18901 consensus_queue.cc:260] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.498649 18901 ts_tablet_manager.cc:1428] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:22.498679 18612 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
W20260927 02:56:22.538658 18747 tablet.cc:2369] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:56:22.557842 18905 raft_consensus.cc:482] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:56:22.557952 18905 raft_consensus.cc:504] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.558254 18905 leader_election.cc:288] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fd90a0645db4448bb9d7fe07b4c3bd7 (127.0.98.3:37403), 6613dc6c96404c04a0c172ac3610e01f (127.0.98.2:36193)
I20260927 02:56:22.561903 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.558630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33611 (local address 127.0.98.2:36193)
0927 02:56:22.558672 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.558674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.558743 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:56:22.558813 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.558814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.558825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.558876 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:22.558973 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.559645 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.560976 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.561558 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.561620 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.561629 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.561796 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.561836 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:56:22.561921 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.558505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52087 (local address 127.0.98.3:37403)
0927 02:56:22.558574 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.558576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.558584 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:22.558609 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.558610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.558619 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.558663 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:22.558828 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.559554 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.560973 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.561550 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.561620 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.561629 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.561816 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.561867 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:56:22.561921 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.558405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37403 (local address 127.0.98.1:52087)
0927 02:56:22.558457 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.558472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.558522 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.558680 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.558681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.558684 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:22.558771 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.558773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.559634 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.559635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.560895 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.560898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.561621 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.561622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.561746 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.561748 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.561795 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.561824 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":40}
I20260927 02:56:22.561928 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.558508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36193 (local address 127.0.98.1:33611)
0927 02:56:22.558676 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.558686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.558744 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.558877 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.558878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.558879 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:22.558924 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.558925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.559709 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.559710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.560896 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.560898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.561621 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.561622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.561723 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.561727 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.561787 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.561824 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":25,"thread_start_us":110,"threads_started":1}
I20260927 02:56:22.562258 18834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8470efbe9cf42c39fc7aaeee224c52f" candidate_uuid: "810fcd53a6b84dd895d63f15f7846eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" is_pre_election: true
I20260927 02:56:22.562384 18834 raft_consensus.cc:2454] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 810fcd53a6b84dd895d63f15f7846eb7 in term 0.
I20260927 02:56:22.562570 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8470efbe9cf42c39fc7aaeee224c52f" candidate_uuid: "810fcd53a6b84dd895d63f15f7846eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6613dc6c96404c04a0c172ac3610e01f" is_pre_election: true
I20260927 02:56:22.562601 18499 leader_election.cc:302] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd90a0645db4448bb9d7fe07b4c3bd7, 810fcd53a6b84dd895d63f15f7846eb7; no voters:
I20260927 02:56:22.562721 18700 raft_consensus.cc:2454] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 810fcd53a6b84dd895d63f15f7846eb7 in term 0.
I20260927 02:56:22.562749 18905 raft_consensus.cc:2781] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:56:22.562842 18905 raft_consensus.cc:482] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:56:22.562872 18905 raft_consensus.cc:3037] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:22.563499 18905 raft_consensus.cc:504] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.563611 18905 leader_election.cc:288] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [CANDIDATE]: Term 1 election: Requested vote from peers 1fd90a0645db4448bb9d7fe07b4c3bd7 (127.0.98.3:37403), 6613dc6c96404c04a0c172ac3610e01f (127.0.98.2:36193)
I20260927 02:56:22.563750 18834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8470efbe9cf42c39fc7aaeee224c52f" candidate_uuid: "810fcd53a6b84dd895d63f15f7846eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7"
I20260927 02:56:22.563791 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8470efbe9cf42c39fc7aaeee224c52f" candidate_uuid: "810fcd53a6b84dd895d63f15f7846eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6613dc6c96404c04a0c172ac3610e01f"
I20260927 02:56:22.563834 18834 raft_consensus.cc:3037] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:22.563854 18700 raft_consensus.cc:3037] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:22.564657 18700 raft_consensus.cc:2454] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 810fcd53a6b84dd895d63f15f7846eb7 in term 1.
I20260927 02:56:22.564657 18834 raft_consensus.cc:2454] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 810fcd53a6b84dd895d63f15f7846eb7 in term 1.
I20260927 02:56:22.564805 18499 leader_election.cc:302] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd90a0645db4448bb9d7fe07b4c3bd7, 810fcd53a6b84dd895d63f15f7846eb7; no voters:
I20260927 02:56:22.564888 18905 raft_consensus.cc:2781] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:22.565028 18905 raft_consensus.cc:686] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [term 1 LEADER]: Becoming Leader. State: Replica: 810fcd53a6b84dd895d63f15f7846eb7, State: Running, Role: LEADER
I20260927 02:56:22.565102 18905 consensus_queue.cc:237] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } }
I20260927 02:56:22.565866 18421 catalog_manager.cc:5495] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 810fcd53a6b84dd895d63f15f7846eb7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "810fcd53a6b84dd895d63f15f7846eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810fcd53a6b84dd895d63f15f7846eb7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } health_report { overall_health: UNKNOWN } } }
I20260927 02:56:22.578564 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.576518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.0.98.1:36613)
0927 02:56:22.576571 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.576572 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.576579 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:22.576608 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.576609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.576623 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.576680 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:22.576781 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.577227 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.577938 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.578136 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.578187 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.578300 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:22.578382 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:22.578447 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.578473 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.578481 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:56:22.580169 18700 raft_consensus.cc:1261] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 1 FOLLOWER]: Refusing update from remote peer 810fcd53a6b84dd895d63f15f7846eb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:22.580169 18834 raft_consensus.cc:1261] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 1 FOLLOWER]: Refusing update from remote peer 810fcd53a6b84dd895d63f15f7846eb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:22.580497 18905 consensus_queue.cc:1035] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd90a0645db4448bb9d7fe07b4c3bd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:22.580619 18907 consensus_queue.cc:1035] T a8470efbe9cf42c39fc7aaeee224c52f P 810fcd53a6b84dd895d63f15f7846eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6613dc6c96404c04a0c172ac3610e01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:22.583396 18911 mvcc.cc:205] Tried to move back new op lower bound from 7333796997445959680 to 7333796997387165696. Current Snapshot: MvccSnapshot[applied={T|T < 7333796997445959680}]
I20260927 02:56:22.584772 18421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:22.584862 18421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:22.585475 18421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0079349fdf9d4976b73720fd56339670: Sending DeleteTablet for 3 replicas of tablet a8470efbe9cf42c39fc7aaeee224c52f
I20260927 02:56:22.585745 18546 tablet_service.cc:1514] Processing DeleteTablet for tablet a8470efbe9cf42c39fc7aaeee224c52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56156
I20260927 02:56:22.585850 18814 tablet_service.cc:1514] Processing DeleteTablet for tablet a8470efbe9cf42c39fc7aaeee224c52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42996
I20260927 02:56:22.585981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 18483
I20260927 02:56:22.586051 18680 tablet_service.cc:1514] Processing DeleteTablet for tablet a8470efbe9cf42c39fc7aaeee224c52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59090
I20260927 02:56:22.587368 18921 tablet_replica.cc:330] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: stopping tablet replica
I20260927 02:56:22.587469 18921 raft_consensus.cc:2229] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:22.587545 18920 tablet_replica.cc:330] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: stopping tablet replica
I20260927 02:56:22.587570 18921 raft_consensus.cc:2258] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:22.587630 18920 raft_consensus.cc:2229] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:22.587716 18920 raft_consensus.cc:2258] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:22.587996 18921 ts_tablet_manager.cc:1905] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:22.588074 18920 ts_tablet_manager.cc:1905] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:22.589406 18921 ts_tablet_manager.cc:1918] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:22.589471 18921 log.cc:1196] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/a8470efbe9cf42c39fc7aaeee224c52f
I20260927 02:56:22.589504 18920 ts_tablet_manager.cc:1918] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:22.589568 18920 log.cc:1196] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/a8470efbe9cf42c39fc7aaeee224c52f
I20260927 02:56:22.589745 18921 ts_tablet_manager.cc:1939] T a8470efbe9cf42c39fc7aaeee224c52f P 6613dc6c96404c04a0c172ac3610e01f: Deleting consensus metadata
I20260927 02:56:22.589836 18920 ts_tablet_manager.cc:1939] T a8470efbe9cf42c39fc7aaeee224c52f P 1fd90a0645db4448bb9d7fe07b4c3bd7: Deleting consensus metadata
I20260927 02:56:22.590128 18405 catalog_manager.cc:4841] TS 6613dc6c96404c04a0c172ac3610e01f (127.0.98.2:36193): tablet a8470efbe9cf42c39fc7aaeee224c52f (table test-table [id=c585381c5f5e416aac0a2b244bf12376]) successfully deleted
I20260927 02:56:22.590301 18405 catalog_manager.cc:4841] TS 1fd90a0645db4448bb9d7fe07b4c3bd7 (127.0.98.3:37403): tablet a8470efbe9cf42c39fc7aaeee224c52f (table test-table [id=c585381c5f5e416aac0a2b244bf12376]) successfully deleted
W20260927 02:56:22.591410 18408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36613 (error 108)
I20260927 02:56:22.591856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 18617
I20260927 02:56:22.591883 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.591771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36613 (local address 127.0.0.1:56188)
0927 02:56:22.591821 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.591859 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:56:22.592579 18408 catalog_manager.cc:4584] TS 810fcd53a6b84dd895d63f15f7846eb7 (127.0.98.1:36613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8470efbe9cf42c39fc7aaeee224c52f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36613: connect: Connection refused (error 111)
I20260927 02:56:22.596648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 18751
I20260927 02:56:22.600132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 18389
2026-09-27T02:56:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1810 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T02:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:22Z Disabled control of system clock
I20260927 02:56:22.624426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39523
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:22.700070 18926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:22.700261 18926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:22.700282 18926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:22.701396 18926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:22.701432 18926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:22.701444 18926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:22.701462 18926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:22.701485 18926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:22.702744 18926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:22.702973 18926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:22.703217 18926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:22.707458 18931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.707489 18932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:22.707626 18926 server_base.cc:1034] running on GCE node
W20260927 02:56:22.707487 18934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:22.707913 18926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:22.708182 18926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:22.709316 18926 hybrid_clock.cc:648] HybridClock initialized: now 1790477782709306 us; error 31 us; skew 500 ppm
I20260927 02:56:22.710368 18926 webserver.cc:460] Webserver started at http://127.0.98.62:36991/ using document root <none> and password file <none>
I20260927 02:56:22.710564 18926 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:22.710609 18926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:22.710729 18926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:22.711697 18926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "ee9251164419403180dc0fd424e0be33"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "09d40e1ef5f38f6ab2a037b11ddeb8a4"
server_key_iv: "e4f75ae6f1c53942fecf37b4a9bdbc00"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.712014 18926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee9251164419403180dc0fd424e0be33"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "09d40e1ef5f38f6ab2a037b11ddeb8a4"
server_key_iv: "e4f75ae6f1c53942fecf37b4a9bdbc00"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.713105 18926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:22.713683 18940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.713828 18926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:22.713912 18926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "ee9251164419403180dc0fd424e0be33"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "09d40e1ef5f38f6ab2a037b11ddeb8a4"
server_key_iv: "e4f75ae6f1c53942fecf37b4a9bdbc00"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.713976 18926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:22.726001 18926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:22.726258 18926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:22.726384 18926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:22.730221 18926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39523
I20260927 02:56:22.730245 18992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39523 every 8 connection(s)
I20260927 02:56:22.730540 18926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:22.731391 18993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:22.733389 18993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33: Bootstrap starting.
I20260927 02:56:22.733897 18993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:22.734139 18993 log.cc:824] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:22.734722 18993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33: No bootstrap required, opened a new log
I20260927 02:56:22.736207 18993 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9251164419403180dc0fd424e0be33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260927 02:56:22.736330 18993 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:22.736352 18993 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9251164419403180dc0fd424e0be33, State: Initialized, Role: FOLLOWER
I20260927 02:56:22.736477 18993 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9251164419403180dc0fd424e0be33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260927 02:56:22.736533 18993 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:22.736554 18993 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:22.736595 18993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:22.737039 18993 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9251164419403180dc0fd424e0be33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260927 02:56:22.737123 18993 leader_election.cc:302] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee9251164419403180dc0fd424e0be33; no voters:
I20260927 02:56:22.737272 18993 leader_election.cc:288] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:22.737318 18996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:22.737489 18993 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:22.737497 18996 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [term 1 LEADER]: Becoming Leader. State: Replica: ee9251164419403180dc0fd424e0be33, State: Running, Role: LEADER
I20260927 02:56:22.737771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 18926
I20260927 02:56:22.737865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:22.738050 392 external_mini_cluster.cc:1428] Setting key 23fe2434dfd9a540988a1d9b37f4928e
I20260927 02:56:22.738267 18996 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9251164419403180dc0fd424e0be33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260927 02:56:22.738826 18998 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9251164419403180dc0fd424e0be33. Latest consensus state: current_term: 1 leader_uuid: "ee9251164419403180dc0fd424e0be33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9251164419403180dc0fd424e0be33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } } }
I20260927 02:56:22.739040 18998 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:22.738890 18997 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee9251164419403180dc0fd424e0be33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9251164419403180dc0fd424e0be33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } } }
I20260927 02:56:22.739128 18997 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33 [sys.catalog]: This master's current role is: LEADER
W20260927 02:56:22.739444 19011 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:22.739542 19011 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:22.739617 19013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:22.739777 19013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:22.741298 19013 catalog_manager.cc:1347] Generated new cluster ID: c559e4efbaf846fbb1cc72bb8338fb9e
I20260927 02:56:22.741354 19013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:22.744372 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.739013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.0.98.62:39523)
0927 02:56:22.739348 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.739356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.739388 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:56:22.739500 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.739506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.739575 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.739779 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:22.740004 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.741176 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.742316 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.742927 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.743029 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.743709 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:22.743713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:22.743713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:22.743909 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:22.743949 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:22.743963 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:22.744031 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.744100 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.744169 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":64,"threads_started":1}
I20260927 02:56:22.753054 19013 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:22.753507 19013 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:22.756315 19013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33: Generated new TSK 0
I20260927 02:56:22.756445 19013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:22.763859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--builtin_ntp_servers=127.0.98.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:22.839884 19017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:22.840063 19017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:22.840082 19017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:22.841189 19017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:22.841225 19017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:22.841248 19017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:56:22.842522 19017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:22.842718 19017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:22.842996 19017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:22.848600 19022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.848585 19025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.848694 19023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:22.848896 19017 server_base.cc:1034] running on GCE node
I20260927 02:56:22.849115 19017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:22.849372 19017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:22.850544 19017 hybrid_clock.cc:648] HybridClock initialized: now 1790477782850526 us; error 30 us; skew 500 ppm
I20260927 02:56:22.851996 19017 webserver.cc:460] Webserver started at http://127.0.98.1:46053/ using document root <none> and password file <none>
I20260927 02:56:22.852284 19017 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:22.852344 19017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:22.852463 19017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:22.853720 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "35e67642fcdb4720a88b7974151060e7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "b3ac7505696761371c4bfd70b012fe29"
server_key_iv: "5f8e4275dd40cc0a1e38744267078ef9"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.854069 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35e67642fcdb4720a88b7974151060e7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "b3ac7505696761371c4bfd70b012fe29"
server_key_iv: "5f8e4275dd40cc0a1e38744267078ef9"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.855530 19017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:22.856274 19031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.856417 19017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:22.856498 19017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "35e67642fcdb4720a88b7974151060e7"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "b3ac7505696761371c4bfd70b012fe29"
server_key_iv: "5f8e4275dd40cc0a1e38744267078ef9"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.856559 19017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:22.865003 19017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:22.865213 19017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:22.865307 19017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:22.865463 19017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:22.865684 19017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:22.865712 19017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.865736 19017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:22.865756 19017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.871021 19017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35085
I20260927 02:56:22.871057 19144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35085 every 8 connection(s)
I20260927 02:56:22.871340 19017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:56:22.875617 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.871946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53415 (local address 127.0.98.62:39523)
0927 02:56:22.871987 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.871989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.872202 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:56:22.872632 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.872632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.872641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.872692 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:22.873248 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.873712 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.874568 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.874741 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.874809 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.875428 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:22.875429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:22.875430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:22.875436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:22.875448 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:22.875452 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:22.875490 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.875541 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.875564 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:56:22.875768 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.871928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39523 (local address 127.0.98.1:53415)
0927 02:56:22.872094 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.872129 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.872560 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.872740 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.872744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.872753 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:22.873192 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.873195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.873776 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.873777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.874510 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.874512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.874816 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.874816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.874925 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.875315 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:22.875316 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:22.875341 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:22.875360 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:22.875490 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:22.875493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:22.875495 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.875535 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.875602 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260927 02:56:22.876199 19145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39523
I20260927 02:56:22.876281 19145 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:22.876441 19145 heartbeater.cc:511] Master 127.0.98.62:39523 requested a full tablet report, sending...
I20260927 02:56:22.876868 18957 ts_manager.cc:194] Registered new tserver with Master: 35e67642fcdb4720a88b7974151060e7 (127.0.98.1:35085)
I20260927 02:56:22.877451 18957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53415
I20260927 02:56:22.878155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 19017
I20260927 02:56:22.878242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:56:22.878381 392 external_mini_cluster.cc:1428] Setting key 99865f2f434d4b1d3661d75a9a38d403
I20260927 02:56:22.879132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--builtin_ntp_servers=127.0.98.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:22.955030 19148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:22.955209 19148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:22.955238 19148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:22.956353 19148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:22.956391 19148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:22.956423 19148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:56:22.957691 19148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:22.957909 19148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:22.958158 19148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:22.962743 19156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.962747 19153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:22.962747 19154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:22.963007 19148 server_base.cc:1034] running on GCE node
I20260927 02:56:22.963181 19148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:22.963359 19148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:22.964481 19148 hybrid_clock.cc:648] HybridClock initialized: now 1790477782964473 us; error 26 us; skew 500 ppm
I20260927 02:56:22.965595 19148 webserver.cc:460] Webserver started at http://127.0.98.2:38301/ using document root <none> and password file <none>
I20260927 02:56:22.965785 19148 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:22.965828 19148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:22.965967 19148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:22.966982 19148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "dc198a28fc0a4df59dd109fbb669dc6f"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "e0052646cbb01abdd34f33cf7e87bed0"
server_key_iv: "d193c6204f6f104f0659eb6c584023f1"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.967280 19148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dc198a28fc0a4df59dd109fbb669dc6f"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "e0052646cbb01abdd34f33cf7e87bed0"
server_key_iv: "d193c6204f6f104f0659eb6c584023f1"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.968405 19148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:56:22.969012 19162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.969144 19148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:22.969214 19148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "dc198a28fc0a4df59dd109fbb669dc6f"
format_stamp: "Formatted at 2026-09-27 02:56:22 on dist-test-slave-mfx9"
server_key: "e0052646cbb01abdd34f33cf7e87bed0"
server_key_iv: "d193c6204f6f104f0659eb6c584023f1"
server_key_version: "encryptionkey@0"
I20260927 02:56:22.969275 19148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:22.987243 19148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:22.987555 19148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:22.987738 19148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:22.988029 19148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:22.988370 19148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:22.988404 19148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.988432 19148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:22.988448 19148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:22.993558 19148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37771
I20260927 02:56:22.993609 19275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37771 every 8 connection(s)
I20260927 02:56:22.993904 19148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:56:22.998337 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.994558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50029 (local address 127.0.98.62:39523)
0927 02:56:22.994606 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:22.994609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:22.994826 (+ 217us) server_negotiation.cc:407] Connection header received
0927 02:56:22.995295 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:22.995296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:22.995305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:22.995341 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:22.995936 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.996410 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.997272 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.997450 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.997493 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.998140 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:22.998141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:22.998141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:22.998145 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:22.998158 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:22.998162 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:22.998200 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:22.998257 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:56:22.998280 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:22.998518 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:22.994546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39523 (local address 127.0.98.2:50029)
0927 02:56:22.994712 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:22.994743 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:22.995207 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:22.995399 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:22.995402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:22.995410 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:22.995874 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.995877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.996470 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.996470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.997215 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:22.997217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:22.997501 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:22.997502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:22.997600 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:22.998028 (+ 428us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:22.998029 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:22.998054 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:22.998075 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:22.998206 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:22.998208 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:22.998210 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:56:22.998254 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:56:22.998335 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260927 02:56:22.998940 19276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39523
I20260927 02:56:22.999019 19276 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:22.999154 19276 heartbeater.cc:511] Master 127.0.98.62:39523 requested a full tablet report, sending...
I20260927 02:56:22.999468 18957 ts_manager.cc:194] Registered new tserver with Master: dc198a28fc0a4df59dd109fbb669dc6f (127.0.98.2:37771)
I20260927 02:56:22.999812 18957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50029
I20260927 02:56:23.003000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 19148
I20260927 02:56:23.003080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:56:23.003245 392 external_mini_cluster.cc:1428] Setting key ca2f0c6ce19a3097f96519e554ad94fa
I20260927 02:56:23.004012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--builtin_ntp_servers=127.0.98.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:23.080787 19279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:23.080977 19279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:23.081005 19279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:23.082090 19279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:23.082127 19279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:23.082159 19279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:56:23.083489 19279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:23.083703 19279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:23.083958 19279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:23.088366 19287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:23.088393 19284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:23.088435 19285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:23.088629 19279 server_base.cc:1034] running on GCE node
I20260927 02:56:23.088819 19279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:23.089030 19279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:23.090166 19279 hybrid_clock.cc:648] HybridClock initialized: now 1790477783090153 us; error 25 us; skew 500 ppm
I20260927 02:56:23.091538 19279 webserver.cc:460] Webserver started at http://127.0.98.3:43627/ using document root <none> and password file <none>
I20260927 02:56:23.091753 19279 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:23.091805 19279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:23.091928 19279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:23.092904 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a3d65652a464a6eab79b852ede4c2da"
format_stamp: "Formatted at 2026-09-27 02:56:23 on dist-test-slave-mfx9"
server_key: "62d9e2a90ad6aa93a0585ceaa0f52234"
server_key_iv: "4bb76f5d8cdc3383cb3a5c1da899de97"
server_key_version: "encryptionkey@0"
I20260927 02:56:23.093219 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a3d65652a464a6eab79b852ede4c2da"
format_stamp: "Formatted at 2026-09-27 02:56:23 on dist-test-slave-mfx9"
server_key: "62d9e2a90ad6aa93a0585ceaa0f52234"
server_key_iv: "4bb76f5d8cdc3383cb3a5c1da899de97"
server_key_version: "encryptionkey@0"
I20260927 02:56:23.094362 19279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:23.095077 19293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:23.095212 19279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:23.095296 19279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "8a3d65652a464a6eab79b852ede4c2da"
format_stamp: "Formatted at 2026-09-27 02:56:23 on dist-test-slave-mfx9"
server_key: "62d9e2a90ad6aa93a0585ceaa0f52234"
server_key_iv: "4bb76f5d8cdc3383cb3a5c1da899de97"
server_key_version: "encryptionkey@0"
I20260927 02:56:23.095357 19279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:23.105454 19279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:23.105675 19279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:23.105785 19279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:23.105960 19279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:23.106235 19279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:23.106276 19279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:23.106302 19279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:23.106326 19279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:23.111515 19279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37325
I20260927 02:56:23.111549 19406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37325 every 8 connection(s)
I20260927 02:56:23.111811 19279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:56:23.116226 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.112421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53509 (local address 127.0.98.62:39523)
0927 02:56:23.112468 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:23.112470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:23.112665 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:56:23.113116 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:23.113117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:23.113127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:23.113173 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:23.113744 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.114225 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.115158 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.115346 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.115392 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.116015 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:23.116015 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:23.116016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:23.116019 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:23.116033 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:23.116037 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:23.116073 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:23.116141 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:56:23.116166 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:23.116407 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.112399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39523 (local address 127.0.98.3:53509)
0927 02:56:23.112562 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:23.112594 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:23.113045 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:23.113223 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:23.113226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:23.113239 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:23.113685 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.113689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.114285 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.114286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.115097 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.115100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.115400 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.115401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.115497 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.115900 (+ 403us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:23.115900 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:23.115925 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:23.115945 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:23.116085 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:23.116088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:23.116090 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:56:23.116137 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:56:23.116225 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260927 02:56:23.116804 19407 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39523
I20260927 02:56:23.116887 19407 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:23.117045 19407 heartbeater.cc:511] Master 127.0.98.62:39523 requested a full tablet report, sending...
I20260927 02:56:23.117384 18957 ts_manager.cc:194] Registered new tserver with Master: 8a3d65652a464a6eab79b852ede4c2da (127.0.98.3:37325)
I20260927 02:56:23.117729 18957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53509
I20260927 02:56:23.118016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 19279
I20260927 02:56:23.118112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:56:23.118285 392 external_mini_cluster.cc:1428] Setting key 48f3c88320fc80b98a7276c08adf081e
I20260927 02:56:23.119032 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:56:23.122850 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.120220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.0.98.62:39523)
0927 02:56:23.120255 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:23.120257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:23.120363 (+ 106us) server_negotiation.cc:407] Connection header received
0927 02:56:23.120406 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:23.120406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:23.120429 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:23.120472 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:23.120629 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.121066 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.121767 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.121950 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.122000 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.122588 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:23.122589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:23.122590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:23.122594 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:23.122605 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:23.122609 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:23.122680 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:23.122751 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:56:23.122778 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:56:23.124091 18957 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:56:23.125484 18957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:56:23.131068 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.127272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.0.98.3:37325)
0927 02:56:23.127772 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:23.127775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:23.127783 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:23.127809 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:23.127811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:23.127823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:23.127916 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:23.128093 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.129286 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.130251 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.130701 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.130783 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.130864 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:23.130963 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 02:56:23.130996 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":88,"threads_started":1}
I20260927 02:56:23.131062 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.127204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37325 (local address 127.0.0.1:45398)
0927 02:56:23.127350 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:23.127373 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:23.127425 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:23.127923 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:23.127924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:23.127926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:23.127953 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.127955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.129352 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.129352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.130185 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.130187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.130797 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.130798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.130871 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.130883 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 02:56:23.130950 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:56:23.130997 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":30,"thread_start_us":72,"threads_started":1}
I20260927 02:56:23.131341 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.127204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37771 (local address 127.0.0.1:50040)
0927 02:56:23.127333 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:23.127369 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:23.127426 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:23.127863 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:23.127866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:23.127869 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:23.127922 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.127923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.129203 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.129204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.130226 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.130227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.131147 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.131148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.131206 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.131210 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:56:23.131237 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 02:56:23.131275 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
I20260927 02:56:23.131423 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.127389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35085 (local address 127.0.0.1:46928)
0927 02:56:23.127520 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:23.127528 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:23.127565 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:23.128171 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:23.128172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:23.128174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:23.128202 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.128203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.128993 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.128994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.130196 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.130199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.131227 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.131228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.131278 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.131281 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:23.131331 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:56:23.131371 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260927 02:56:23.131486 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.127741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.0.98.1:35085)
0927 02:56:23.128045 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:23.128049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:23.128057 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:23.128091 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:23.128093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:23.128107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:23.128176 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:23.128249 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.128916 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.130293 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.131051 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.131117 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.131199 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:23.131364 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 02:56:23.131410 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":71,"threads_started":1}
I20260927 02:56:23.131662 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.127528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.0.98.2:37771)
0927 02:56:23.127667 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:23.127671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:23.127682 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:56:23.127718 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:23.127720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:23.127735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:23.127846 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:23.128011 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.129127 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.130306 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.131051 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.131432 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.131525 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:23.131552 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 02:56:23.131587 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260927 02:56:23.131889 19341 tablet_service.cc:1467] Processing CreateTablet for tablet 559f262a30cd471b9c5a2abbace1c61d (DEFAULT_TABLE table=test-table [id=f370dd2ea73a4349976bbe0599d3ba25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:23.132187 19341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559f262a30cd471b9c5a2abbace1c61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:23.132202 19210 tablet_service.cc:1467] Processing CreateTablet for tablet 559f262a30cd471b9c5a2abbace1c61d (DEFAULT_TABLE table=test-table [id=f370dd2ea73a4349976bbe0599d3ba25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:23.132539 19210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559f262a30cd471b9c5a2abbace1c61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:23.134461 19421 tablet_bootstrap.cc:492] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: Bootstrap starting.
I20260927 02:56:23.134940 19422 tablet_bootstrap.cc:492] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Bootstrap starting.
I20260927 02:56:23.134924 19079 tablet_service.cc:1467] Processing CreateTablet for tablet 559f262a30cd471b9c5a2abbace1c61d (DEFAULT_TABLE table=test-table [id=f370dd2ea73a4349976bbe0599d3ba25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:23.135254 19079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559f262a30cd471b9c5a2abbace1c61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:23.135253 19421 tablet_bootstrap.cc:654] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:23.135499 19422 tablet_bootstrap.cc:654] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:23.135514 19421 log.cc:824] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:23.135774 19422 log.cc:824] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:23.136107 19421 tablet_bootstrap.cc:492] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: No bootstrap required, opened a new log
I20260927 02:56:23.136161 19421 ts_tablet_manager.cc:1397] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:23.136467 19422 tablet_bootstrap.cc:492] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: No bootstrap required, opened a new log
I20260927 02:56:23.136521 19422 ts_tablet_manager.cc:1397] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:23.137905 19421 raft_consensus.cc:348] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.138033 19421 raft_consensus.cc:374] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:23.138053 19421 raft_consensus.cc:729] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3d65652a464a6eab79b852ede4c2da, State: Initialized, Role: FOLLOWER
I20260927 02:56:23.138135 19421 consensus_queue.cc:260] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.138315 19425 tablet_bootstrap.cc:492] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7: Bootstrap starting.
I20260927 02:56:23.138414 19421 ts_tablet_manager.cc:1428] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:23.138547 19407 heartbeater.cc:503] Master 127.0.98.62:39523 was elected leader, sending a full tablet report...
I20260927 02:56:23.138885 19422 raft_consensus.cc:348] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.139036 19422 raft_consensus.cc:374] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:23.139061 19422 raft_consensus.cc:729] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc198a28fc0a4df59dd109fbb669dc6f, State: Initialized, Role: FOLLOWER
I20260927 02:56:23.139163 19422 consensus_queue.cc:260] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.139397 19422 ts_tablet_manager.cc:1428] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 02:56:23.139449 19276 heartbeater.cc:503] Master 127.0.98.62:39523 was elected leader, sending a full tablet report...
I20260927 02:56:23.139782 19425 tablet_bootstrap.cc:654] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:23.140072 19425 log.cc:824] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:23.140666 19425 tablet_bootstrap.cc:492] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7: No bootstrap required, opened a new log
I20260927 02:56:23.140717 19425 ts_tablet_manager.cc:1397] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:23.142323 19425 raft_consensus.cc:348] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.142434 19425 raft_consensus.cc:374] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:23.142463 19425 raft_consensus.cc:729] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e67642fcdb4720a88b7974151060e7, State: Initialized, Role: FOLLOWER
I20260927 02:56:23.142565 19425 consensus_queue.cc:260] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.142824 19425 ts_tablet_manager.cc:1428] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:23.142902 19145 heartbeater.cc:503] Master 127.0.98.62:39523 was elected leader, sending a full tablet report...
W20260927 02:56:23.244391 19277 tablet.cc:2369] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:56:23.249022 19429 raft_consensus.cc:482] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:56:23.249161 19429 raft_consensus.cc:504] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.249579 19429 leader_election.cc:288] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a3d65652a464a6eab79b852ede4c2da (127.0.98.3:37325), dc198a28fc0a4df59dd109fbb669dc6f (127.0.98.2:37771)
I20260927 02:56:23.252549 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.249951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32973 (local address 127.0.98.3:37325)
0927 02:56:23.250025 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:23.250026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:23.250036 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:56:23.250066 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:23.250066 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:23.250075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:23.250143 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:23.250341 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.250987 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.251892 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.252243 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.252302 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.252312 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:23.252434 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 02:56:23.252468 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:56:23.252554 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.249750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37325 (local address 127.0.98.1:32973)
0927 02:56:23.249808 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:23.249819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:23.249876 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:23.250172 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:23.250173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:23.250175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:23.250262 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.250264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.251065 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.251066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.251812 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.251814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.252309 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.252310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.252379 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.252382 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:23.252424 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:56:23.252465 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:56:23.252583 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.250117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54589 (local address 127.0.98.2:37771)
0927 02:56:23.250168 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:23.250170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:23.250177 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:23.250200 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:23.250201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:23.250209 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:23.250256 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:23.250426 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.251045 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.251930 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.252283 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.252340 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.252350 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:23.252486 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:56:23.252518 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:56:23.252626 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.249761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37771 (local address 127.0.98.1:54589)
0927 02:56:23.249947 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:23.249956 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:23.250034 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:23.250308 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:23.250309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:23.250310 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:23.250365 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.250367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.251124 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.251125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.251851 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:23.251853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.252359 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.252360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:23.252427 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.252430 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:23.252473 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:56:23.252512 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":115,"threads_started":1}
I20260927 02:56:23.252938 19361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559f262a30cd471b9c5a2abbace1c61d" candidate_uuid: "35e67642fcdb4720a88b7974151060e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a3d65652a464a6eab79b852ede4c2da" is_pre_election: true
I20260927 02:56:23.253080 19361 raft_consensus.cc:2454] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e67642fcdb4720a88b7974151060e7 in term 0.
I20260927 02:56:23.253171 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559f262a30cd471b9c5a2abbace1c61d" candidate_uuid: "35e67642fcdb4720a88b7974151060e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" is_pre_election: true
I20260927 02:56:23.253314 19230 raft_consensus.cc:2454] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e67642fcdb4720a88b7974151060e7 in term 0.
I20260927 02:56:23.253348 19032 leader_election.cc:302] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e67642fcdb4720a88b7974151060e7, 8a3d65652a464a6eab79b852ede4c2da; no voters:
I20260927 02:56:23.253482 19429 raft_consensus.cc:2781] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:56:23.253530 19429 raft_consensus.cc:482] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:56:23.253544 19429 raft_consensus.cc:3037] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:23.254166 19429 raft_consensus.cc:504] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.254274 19429 leader_election.cc:288] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [CANDIDATE]: Term 1 election: Requested vote from peers 8a3d65652a464a6eab79b852ede4c2da (127.0.98.3:37325), dc198a28fc0a4df59dd109fbb669dc6f (127.0.98.2:37771)
I20260927 02:56:23.254428 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559f262a30cd471b9c5a2abbace1c61d" candidate_uuid: "35e67642fcdb4720a88b7974151060e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc198a28fc0a4df59dd109fbb669dc6f"
I20260927 02:56:23.254494 19230 raft_consensus.cc:3037] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:23.255277 19230 raft_consensus.cc:2454] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e67642fcdb4720a88b7974151060e7 in term 1.
I20260927 02:56:23.255404 19361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559f262a30cd471b9c5a2abbace1c61d" candidate_uuid: "35e67642fcdb4720a88b7974151060e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a3d65652a464a6eab79b852ede4c2da"
I20260927 02:56:23.255498 19361 raft_consensus.cc:3037] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:23.256340 19361 raft_consensus.cc:2454] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e67642fcdb4720a88b7974151060e7 in term 1.
I20260927 02:56:23.256492 19032 leader_election.cc:302] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e67642fcdb4720a88b7974151060e7, 8a3d65652a464a6eab79b852ede4c2da; no voters:
I20260927 02:56:23.256596 19429 raft_consensus.cc:2781] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:23.256781 19429 raft_consensus.cc:686] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [term 1 LEADER]: Becoming Leader. State: Replica: 35e67642fcdb4720a88b7974151060e7, State: Running, Role: LEADER
I20260927 02:56:23.256906 19429 consensus_queue.cc:237] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } }
I20260927 02:56:23.257917 18957 catalog_manager.cc:5495] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35e67642fcdb4720a88b7974151060e7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "35e67642fcdb4720a88b7974151060e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35e67642fcdb4720a88b7974151060e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35085 } health_report { overall_health: HEALTHY } } }
I20260927 02:56:23.273041 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.271021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.0.98.1:35085)
0927 02:56:23.271062 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:23.271063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:23.271070 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:23.271122 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:23.271122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:23.271137 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:23.271190 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:23.271274 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.271703 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.272397 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:23.272569 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:23.272611 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:23.272749 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:23.272845 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:23.272928 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:23.272968 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 02:56:23.272977 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:56:23.274564 19230 raft_consensus.cc:1261] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 1 FOLLOWER]: Refusing update from remote peer 35e67642fcdb4720a88b7974151060e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:23.274588 19361 raft_consensus.cc:1261] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 1 FOLLOWER]: Refusing update from remote peer 35e67642fcdb4720a88b7974151060e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:23.274870 19429 consensus_queue.cc:1035] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc198a28fc0a4df59dd109fbb669dc6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:23.275000 19431 consensus_queue.cc:1035] T 559f262a30cd471b9c5a2abbace1c61d P 35e67642fcdb4720a88b7974151060e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a3d65652a464a6eab79b852ede4c2da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:23.277333 19435 mvcc.cc:205] Tried to move back new op lower bound from 7333797000290398208 to 7333797000220815360. Current Snapshot: MvccSnapshot[applied={T|T < 7333797000290398208}]
I20260927 02:56:23.278172 19436 mvcc.cc:205] Tried to move back new op lower bound from 7333797000290398208 to 7333797000220815360. Current Snapshot: MvccSnapshot[applied={T|T < 7333797000290398208}]
I20260927 02:56:23.279654 18957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:23.279740 18957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:23.280366 18957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee9251164419403180dc0fd424e0be33: Sending DeleteTablet for 3 replicas of tablet 559f262a30cd471b9c5a2abbace1c61d
I20260927 02:56:23.280701 19341 tablet_service.cc:1514] Processing DeleteTablet for tablet 559f262a30cd471b9c5a2abbace1c61d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:23 UTC) from {username='slave'} at 127.0.0.1:45398
I20260927 02:56:23.280740 19079 tablet_service.cc:1514] Processing DeleteTablet for tablet 559f262a30cd471b9c5a2abbace1c61d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:23 UTC) from {username='slave'} at 127.0.0.1:46928
I20260927 02:56:23.280777 19210 tablet_service.cc:1514] Processing DeleteTablet for tablet 559f262a30cd471b9c5a2abbace1c61d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:23 UTC) from {username='slave'} at 127.0.0.1:50040
I20260927 02:56:23.280982 19445 tablet_replica.cc:330] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: stopping tablet replica
I20260927 02:56:23.281054 19443 tablet_replica.cc:330] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: stopping tablet replica
I20260927 02:56:23.281113 19445 raft_consensus.cc:2229] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:23.281152 19443 raft_consensus.cc:2229] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:23.281239 19445 raft_consensus.cc:2258] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:23.281294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 19017
I20260927 02:56:23.281251 19443 raft_consensus.cc:2258] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:23.282203 19445 ts_tablet_manager.cc:1905] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:23.282334 19443 ts_tablet_manager.cc:1905] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:23.284250 19445 ts_tablet_manager.cc:1918] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:23.284250 19443 ts_tablet_manager.cc:1918] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:23.284322 19443 log.cc:1196] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/559f262a30cd471b9c5a2abbace1c61d
I20260927 02:56:23.284322 19445 log.cc:1196] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/559f262a30cd471b9c5a2abbace1c61d
I20260927 02:56:23.284554 19443 ts_tablet_manager.cc:1939] T 559f262a30cd471b9c5a2abbace1c61d P 8a3d65652a464a6eab79b852ede4c2da: Deleting consensus metadata
I20260927 02:56:23.284554 19445 ts_tablet_manager.cc:1939] T 559f262a30cd471b9c5a2abbace1c61d P dc198a28fc0a4df59dd109fbb669dc6f: Deleting consensus metadata
I20260927 02:56:23.284910 18943 catalog_manager.cc:4841] TS dc198a28fc0a4df59dd109fbb669dc6f (127.0.98.2:37771): tablet 559f262a30cd471b9c5a2abbace1c61d (table test-table [id=f370dd2ea73a4349976bbe0599d3ba25]) successfully deleted
I20260927 02:56:23.284940 18941 catalog_manager.cc:4841] TS 8a3d65652a464a6eab79b852ede4c2da (127.0.98.3:37325): tablet 559f262a30cd471b9c5a2abbace1c61d (table test-table [id=f370dd2ea73a4349976bbe0599d3ba25]) successfully deleted
W20260927 02:56:23.286782 18943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35085 (error 108)
I20260927 02:56:23.286984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 19148
I20260927 02:56:23.287504 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:23.287382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35085 (local address 127.0.0.1:46942)
0927 02:56:23.287431 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:23.287476 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:56:23.287619 18943 catalog_manager.cc:4584] TS 35e67642fcdb4720a88b7974151060e7 (127.0.98.1:35085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 559f262a30cd471b9c5a2abbace1c61d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35085: connect: Connection refused (error 111)
I20260927 02:56:23.292520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 19279
I20260927 02:56:23.297770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 18926
2026-09-27T02:56:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (697 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:23 dist-test-slave-mfx9 krb5kdc[19449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:23 dist-test-slave-mfx9 krb5kdc[19449](info): set up 2 sockets
Sep 27 02:56:23 dist-test-slave-mfx9 krb5kdc[19449](info): commencing operation
krb5kdc: starting...
W20260927 02:56:24.250288 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:24.354012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41585
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:24.430172 19462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:24.430367 19462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:24.430416 19462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:24.431603 19462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:24.431643 19462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:24.431656 19462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:24.431667 19462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:24.431679 19462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:24.433102 19462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:24.433310 19462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:24.433511 19462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:24.437916 19467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:24.437963 19468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:24.437916 19470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:24.438061 19462 server_base.cc:1034] running on GCE node
I20260927 02:56:24.438328 19462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:24.438556 19462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:24.439684 19462 hybrid_clock.cc:648] HybridClock initialized: now 1790477784439666 us; error 34 us; skew 500 ppm
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:24.441915 19462 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:24.442276 19462 webserver.cc:460] Webserver started at http://127.0.98.62:38201/ using document root <none> and password file <none>
I20260927 02:56:24.442467 19462 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:24.442507 19462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:24.442623 19462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:24.443640 19462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "e61dbe8d3ea3452eaf516f06e90be643"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "19e96eb55525a2ffbf873d08154df7c1"
server_key_iv: "17d80fe2abd76472a098329759d539e2"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.443944 19462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "e61dbe8d3ea3452eaf516f06e90be643"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "19e96eb55525a2ffbf873d08154df7c1"
server_key_iv: "17d80fe2abd76472a098329759d539e2"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.445040 19462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:24.445602 19477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.445724 19462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:24.445799 19462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "e61dbe8d3ea3452eaf516f06e90be643"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "19e96eb55525a2ffbf873d08154df7c1"
server_key_iv: "17d80fe2abd76472a098329759d539e2"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.445869 19462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:24.461530 19462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:24.463588 19462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:24.463717 19462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:24.467306 19462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41585
I20260927 02:56:24.467350 19529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41585 every 8 connection(s)
I20260927 02:56:24.467648 19462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:24.468434 19530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:24.470515 19530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643: Bootstrap starting.
I20260927 02:56:24.471058 19530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:24.471335 19530 log.cc:824] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:24.471930 19530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643: No bootstrap required, opened a new log
I20260927 02:56:24.473409 19530 raft_consensus.cc:348] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61dbe8d3ea3452eaf516f06e90be643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } }
I20260927 02:56:24.473539 19530 raft_consensus.cc:374] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:24.473572 19530 raft_consensus.cc:729] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61dbe8d3ea3452eaf516f06e90be643, State: Initialized, Role: FOLLOWER
I20260927 02:56:24.473702 19530 consensus_queue.cc:260] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61dbe8d3ea3452eaf516f06e90be643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } }
I20260927 02:56:24.473759 19530 raft_consensus.cc:388] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:24.473783 19530 raft_consensus.cc:482] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:24.473827 19530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:24.474293 19530 raft_consensus.cc:504] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61dbe8d3ea3452eaf516f06e90be643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } }
I20260927 02:56:24.474376 19530 leader_election.cc:302] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e61dbe8d3ea3452eaf516f06e90be643; no voters:
I20260927 02:56:24.474520 19530 leader_election.cc:288] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:24.474550 19533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:24.474686 19533 raft_consensus.cc:686] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [term 1 LEADER]: Becoming Leader. State: Replica: e61dbe8d3ea3452eaf516f06e90be643, State: Running, Role: LEADER
I20260927 02:56:24.474726 19530 sys_catalog.cc:564] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:24.474783 19533 consensus_queue.cc:237] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61dbe8d3ea3452eaf516f06e90be643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } }
I20260927 02:56:24.475114 19535 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61dbe8d3ea3452eaf516f06e90be643. Latest consensus state: current_term: 1 leader_uuid: "e61dbe8d3ea3452eaf516f06e90be643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61dbe8d3ea3452eaf516f06e90be643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } }
I20260927 02:56:24.475190 19535 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:24.475318 19534 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e61dbe8d3ea3452eaf516f06e90be643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61dbe8d3ea3452eaf516f06e90be643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } }
I20260927 02:56:24.475426 19534 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:24.475410 19541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:24.476089 19541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:24.477520 19541 catalog_manager.cc:1347] Generated new cluster ID: 8f8473ccce8a461b8e1b06e48c76896c
I20260927 02:56:24.477574 19541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:24.477743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 19462
I20260927 02:56:24.477843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:24.478009 392 external_mini_cluster.cc:1428] Setting key 33c3449f7f0f88d595ad17223f67ddeb
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:24.489293 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.478548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.0.98.62:41585)
0927 02:56:24.478857 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:24.478867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:24.478901 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:56:24.479132 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:24.479140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:24.479209 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:24.479339 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:24.480054 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.481433 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.482493 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.483074 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.483159 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.485789 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:24.485793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:24.485793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:24.485804 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:24.487667 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.488018 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.488021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.488021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.488122 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.488285 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.488286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.488286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.488585 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:24.488725 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:24.488818 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 02:56:24.489055 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":69,"threads_started":1}
I20260927 02:56:24.509967 19541 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:24.510433 19541 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:24.516721 19541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643: Generated new TSK 0
I20260927 02:56:24.516885 19541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:56:24.545205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41585
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:24.636459 19556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:24.636639 19556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:24.636659 19556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:24.637758 19556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:24.637794 19556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:24.637821 19556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:56:24.639256 19556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:24.639465 19556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:24.639654 19556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:24.644023 19562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:24.644029 19561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:24.644232 19556 server_base.cc:1034] running on GCE node
W20260927 02:56:24.644054 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:24.644440 19556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:24.644639 19556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:24.645778 19556 hybrid_clock.cc:648] HybridClock initialized: now 1790477784645759 us; error 31 us; skew 500 ppm
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:24.647953 19556 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:56:24.648375 19556 webserver.cc:460] Webserver started at http://127.0.98.1:44309/ using document root <none> and password file <none>
I20260927 02:56:24.648562 19556 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:24.648607 19556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:24.648725 19556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:24.649693 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "7522307c282d499ba664a7dbd5552153"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "70b57fd42171b9732c8a940904fabf2e"
server_key_iv: "5a06d971e95227793949cddd61be0c45"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.650000 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7522307c282d499ba664a7dbd5552153"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "70b57fd42171b9732c8a940904fabf2e"
server_key_iv: "5a06d971e95227793949cddd61be0c45"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.651154 19556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:24.651719 19571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.651851 19556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:24.651926 19556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "7522307c282d499ba664a7dbd5552153"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "70b57fd42171b9732c8a940904fabf2e"
server_key_iv: "5a06d971e95227793949cddd61be0c45"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.651980 19556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:24.664197 19556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:24.665648 19556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:24.665773 19556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:24.665946 19556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:24.666193 19556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:24.666222 19556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.666247 19556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:24.666263 19556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.671450 19556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40423
I20260927 02:56:24.671504 19684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40423 every 8 connection(s)
I20260927 02:56:24.671753 19556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:24.679934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 19556
I20260927 02:56:24.679991 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.672377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55367 (local address 127.0.98.62:41585)
0927 02:56:24.672428 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:24.672430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:24.672591 (+ 161us) server_negotiation.cc:407] Connection header received
0927 02:56:24.673468 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:24.673469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:24.673485 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:24.673536 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:24.674096 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.674599 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.675639 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.675859 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.675912 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.677757 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:24.677759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:24.677759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:24.677766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:24.678839 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.679169 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.679170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.679170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.679224 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.679423 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.679424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.679424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.679531 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:24.679585 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:24.679818 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:56:24.679904 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:56:24.680037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:56:24.680177 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.672354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41585 (local address 127.0.98.1:55367)
0927 02:56:24.672486 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:24.672519 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:24.673369 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:24.673601 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:24.673604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:24.673829 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:24.674008 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.674012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.674677 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.674678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.675557 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.675560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.675925 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.675926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.676043 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.676466 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:24.676473 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:24.677675 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:24.678900 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:24.678907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:24.678909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:24.679106 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:24.679274 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:24.679274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:24.679275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:24.679368 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:24.679581 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:24.679587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:24.679708 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 02:56:24.679792 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:56:24.679940 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260927 02:56:24.680227 392 external_mini_cluster.cc:1428] Setting key 5a9f55fe0b5b935906a0be232ed09504
I20260927 02:56:24.680783 19685 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41585
I20260927 02:56:24.680886 19685 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:24.681056 19685 heartbeater.cc:511] Master 127.0.98.62:41585 requested a full tablet report, sending...
I20260927 02:56:24.681561 19494 ts_manager.cc:194] Registered new tserver with Master: 7522307c282d499ba664a7dbd5552153 (127.0.98.1:40423)
I20260927 02:56:24.682279 19494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55367
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:56:24.701582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41585
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:24.779172 19690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:24.779345 19690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:24.779366 19690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:24.780444 19690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:24.780476 19690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:24.780499 19690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:56:24.781872 19690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:24.782038 19690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:24.782193 19690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:24.786562 19698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:24.786598 19696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:24.786603 19695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:24.786909 19690 server_base.cc:1034] running on GCE node
I20260927 02:56:24.787065 19690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:24.787264 19690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:24.788394 19690 hybrid_clock.cc:648] HybridClock initialized: now 1790477784788379 us; error 27 us; skew 500 ppm
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:24.790522 19690 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:56:24.790952 19690 webserver.cc:460] Webserver started at http://127.0.98.2:45563/ using document root <none> and password file <none>
I20260927 02:56:24.791147 19690 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:24.791191 19690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:24.791308 19690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:24.792240 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "4808cc9659fa41e5a80e390f68588027"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "80919b5af9c84b2b49afa937eb8ab8cb"
server_key_iv: "3fde53efc9563ea09e40b020d7ed93d2"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.792547 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4808cc9659fa41e5a80e390f68588027"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "80919b5af9c84b2b49afa937eb8ab8cb"
server_key_iv: "3fde53efc9563ea09e40b020d7ed93d2"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.793654 19690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:24.794240 19705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.794389 19690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:24.794462 19690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "4808cc9659fa41e5a80e390f68588027"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "80919b5af9c84b2b49afa937eb8ab8cb"
server_key_iv: "3fde53efc9563ea09e40b020d7ed93d2"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.794525 19690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:24.807906 19690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:24.809392 19690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:24.809511 19690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:24.809680 19690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:24.809934 19690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:24.809966 19690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.809991 19690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:24.810014 19690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.815162 19690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42767
I20260927 02:56:24.815197 19818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42767 every 8 connection(s)
I20260927 02:56:24.815495 19690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:24.823355 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.816095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37957 (local address 127.0.98.62:41585)
0927 02:56:24.816168 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:24.816170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:24.816327 (+ 157us) server_negotiation.cc:407] Connection header received
0927 02:56:24.817140 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:24.817141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:24.817151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:24.817196 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:24.817711 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.818190 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.819138 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.819315 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.819369 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.821172 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:24.821173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:24.821173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:24.821179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:24.822359 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.822655 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.822656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.822656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.822698 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.822898 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.822899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.822899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.822994 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:24.823043 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:24.823230 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 02:56:24.823287 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:56:24.823501 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.816066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41585 (local address 127.0.98.2:37957)
0927 02:56:24.816215 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:24.816247 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:24.817062 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:24.817251 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:24.817254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:24.817479 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:24.817653 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.817657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.818247 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.818247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.819083 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.819085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.819374 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.819375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.819463 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.819841 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:24.819848 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:24.821089 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:24.822413 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:24.822418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:24.822420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:24.822604 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:24.822736 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:24.822737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:24.822737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:24.822853 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:24.823041 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:24.823043 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:24.823142 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 02:56:24.823214 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:56:24.823331 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260927 02:56:24.823899 19819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41585
I20260927 02:56:24.823979 19819 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:24.824136 19819 heartbeater.cc:511] Master 127.0.98.62:41585 requested a full tablet report, sending...
I20260927 02:56:24.824443 19494 ts_manager.cc:194] Registered new tserver with Master: 4808cc9659fa41e5a80e390f68588027 (127.0.98.2:42767)
I20260927 02:56:24.824770 19494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37957
I20260927 02:56:24.825515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 19690
I20260927 02:56:24.825621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:56:24.825824 392 external_mini_cluster.cc:1428] Setting key aabbb170d3e261016385831dc1a092e1
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:56:24.847589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41585
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:24.924598 19824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:24.924795 19824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:24.924818 19824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:24.925930 19824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:24.925972 19824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:24.926004 19824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:56:24.927430 19824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:24.927637 19824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:24.927842 19824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:56:24.932497 19824 server_base.cc:1034] running on GCE node
W20260927 02:56:24.932468 19832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:24.932475 19830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:24.932463 19829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:24.932827 19824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:24.933043 19824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:24.934180 19824 hybrid_clock.cc:648] HybridClock initialized: now 1790477784934167 us; error 33 us; skew 500 ppm
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:24.936472 19824 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:56:24.936848 19824 webserver.cc:460] Webserver started at http://127.0.98.3:38631/ using document root <none> and password file <none>
I20260927 02:56:24.937022 19824 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:24.937067 19824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:24.937180 19824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:24.938109 19824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9dcbee629874a609665979a0b357bf6"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "663230e314eef037b10c81c82da942e0"
server_key_iv: "6db6bee4db540bcb661620355edec622"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.938418 19824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9dcbee629874a609665979a0b357bf6"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "663230e314eef037b10c81c82da942e0"
server_key_iv: "6db6bee4db540bcb661620355edec622"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.939586 19824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:24.940173 19839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.940313 19824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:24.940397 19824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "f9dcbee629874a609665979a0b357bf6"
format_stamp: "Formatted at 2026-09-27 02:56:24 on dist-test-slave-mfx9"
server_key: "663230e314eef037b10c81c82da942e0"
server_key_iv: "6db6bee4db540bcb661620355edec622"
server_key_version: "encryptionkey@0"
I20260927 02:56:24.940462 19824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:24.952689 19824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:24.954563 19824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:24.954674 19824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:24.954890 19824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:24.955134 19824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:24.955165 19824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.955191 19824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:24.955214 19824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:24.960338 19824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42589
I20260927 02:56:24.960400 19952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42589 every 8 connection(s)
I20260927 02:56:24.960649 19824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:56:24.961427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 19824
I20260927 02:56:24.961519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:56:24.961679 392 external_mini_cluster.cc:1428] Setting key 4c181ac93ec4da1d9b26abe2078368ca
Sep 27 02:56:24 dist-test-slave-mfx9 krb5kdc[19449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:24.968859 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.961272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36011 (local address 127.0.98.62:41585)
0927 02:56:24.961342 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:24.961344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:24.961487 (+ 143us) server_negotiation.cc:407] Connection header received
0927 02:56:24.962522 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:24.962523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:24.962534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:24.962579 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:24.963183 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.963676 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.964638 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.964839 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.964891 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.966647 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:24.966648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:24.966649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:24.966655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:24.967698 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.968039 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.968040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.968040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.968091 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.968291 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.968292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.968292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.968405 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:24.968457 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:24.968699 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 02:56:24.968772 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:56:24.969022 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.961233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41585 (local address 127.0.98.3:36011)
0927 02:56:24.961382 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:24.961419 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:24.962438 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:24.962648 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:24.962652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:24.962931 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:24.963104 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.963108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.963751 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.963752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.964563 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.964565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.964903 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.964904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.965029 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.965384 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:24.965391 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:24.966568 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:24.967766 (+ 1198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:24.967772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:24.967774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:24.967977 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:24.968142 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:24.968143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:24.968143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:24.968238 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:24.968457 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:24.968463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:24.968587 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 02:56:24.968669 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:56:24.968817 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":98,"threads_started":1}
I20260927 02:56:24.969411 19953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41585
I20260927 02:56:24.969484 19953 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:24.969612 19953 heartbeater.cc:511] Master 127.0.98.62:41585 requested a full tablet report, sending...
I20260927 02:56:24.969905 19494 ts_manager.cc:194] Registered new tserver with Master: f9dcbee629874a609665979a0b357bf6 (127.0.98.3:42589)
I20260927 02:56:24.970216 19494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36011
I20260927 02:56:24.973115 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:56:24.979473 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.974285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.0.98.62:41585)
0927 02:56:24.974344 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:24.974345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:24.974431 (+ 86us) server_negotiation.cc:407] Connection header received
0927 02:56:24.974534 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:24.974534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:24.974543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:24.974590 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:24.975117 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.975651 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.976375 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.976572 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.976620 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.977538 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:24.977539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:24.977539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:24.977544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:24.978664 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.978957 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.978958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.978958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.979000 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:24.979120 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:24.979120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:24.979120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:24.979229 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:24.979301 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:24.979368 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:56:24.979419 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:56:24.980808 19494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:56:24.982255 19494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:56:24.988247 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.984143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40423 (local address 127.0.0.1:47790)
0927 02:56:24.984458 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:24.984466 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:24.984505 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:24.984743 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:24.984745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:24.984747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:24.984790 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.984791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.986267 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.986268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.987162 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.987165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.988019 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.988019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.988081 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.988090 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 02:56:24.988141 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:56:24.988177 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":4,"thread_start_us":54,"threads_started":1}
I20260927 02:56:24.988282 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.984240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.0.98.1:40423)
0927 02:56:24.984425 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:24.984431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:24.984500 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:56:24.984586 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:24.984588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:24.984603 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:24.984652 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:24.984864 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.985965 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.987243 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.987937 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.988023 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.988083 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:24.988158 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 02:56:24.988200 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
I20260927 02:56:24.988593 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.984143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42589 (local address 127.0.0.1:44818)
0927 02:56:24.984254 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:24.984295 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:24.984368 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:24.984759 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:24.984760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:24.984763 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:24.984827 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.984830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.986062 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.986063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.987398 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.987400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.988215 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.988215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.988269 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.988272 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:24.988493 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 02:56:24.988532 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":70,"thread_start_us":59,"threads_started":1}
I20260927 02:56:24.988862 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.984426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.0.98.3:42589)
0927 02:56:24.984598 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:24.984602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:24.984613 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:56:24.984648 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:24.984649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:24.984664 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:24.984750 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:24.984920 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.985952 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.987478 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.988127 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.988208 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.988262 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:24.988752 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 02:56:24.988792 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260927 02:56:24.989089 19619 tablet_service.cc:1467] Processing CreateTablet for tablet 28cbc0ee179b4921a7190b7fe1db43e9 (DEFAULT_TABLE table=test-table [id=752875552976405280490aff86970222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:24.989384 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.984328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42767 (local address 127.0.0.1:60590)
0927 02:56:24.984842 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:24.984852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:24.984900 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:24.985251 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:24.985252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:24.985254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:24.985285 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.985286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.986122 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.986124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.987394 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:24.987398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.989137 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.989138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:24.989212 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.989217 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:56:24.989272 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:56:24.989313 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":39,"thread_start_us":37,"threads_started":1}
I20260927 02:56:24.989423 19619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28cbc0ee179b4921a7190b7fe1db43e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:24.989413 19887 tablet_service.cc:1467] Processing CreateTablet for tablet 28cbc0ee179b4921a7190b7fe1db43e9 (DEFAULT_TABLE table=test-table [id=752875552976405280490aff86970222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:24.989670 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:24.984452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.98.2:42767)
0927 02:56:24.984609 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:24.984613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:24.985013 (+ 400us) server_negotiation.cc:407] Connection header received
0927 02:56:24.985061 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:24.985063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:24.985078 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:24.985149 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:24.985357 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.986022 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.988299 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:24.988969 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:24.989043 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:24.989100 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:24.989541 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 02:56:24.989588 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260927 02:56:24.989699 19887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28cbc0ee179b4921a7190b7fe1db43e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:24.990243 19753 tablet_service.cc:1467] Processing CreateTablet for tablet 28cbc0ee179b4921a7190b7fe1db43e9 (DEFAULT_TABLE table=test-table [id=752875552976405280490aff86970222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:24.990523 19753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28cbc0ee179b4921a7190b7fe1db43e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:24.992892 19968 tablet_bootstrap.cc:492] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: Bootstrap starting.
I20260927 02:56:24.993111 19969 tablet_bootstrap.cc:492] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027: Bootstrap starting.
I20260927 02:56:24.993511 19968 tablet_bootstrap.cc:654] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:24.993665 19969 tablet_bootstrap.cc:654] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:24.994098 19968 log.cc:824] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:24.994174 19969 log.cc:824] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:24.994421 19967 tablet_bootstrap.cc:492] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153: Bootstrap starting.
I20260927 02:56:24.994798 19969 tablet_bootstrap.cc:492] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027: No bootstrap required, opened a new log
I20260927 02:56:24.994831 19968 tablet_bootstrap.cc:492] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: No bootstrap required, opened a new log
I20260927 02:56:24.994869 19969 ts_tablet_manager.cc:1397] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:24.994879 19968 ts_tablet_manager.cc:1397] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:24.994968 19967 tablet_bootstrap.cc:654] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:24.995409 19967 log.cc:824] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:24.996034 19967 tablet_bootstrap.cc:492] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153: No bootstrap required, opened a new log
I20260927 02:56:24.996098 19967 ts_tablet_manager.cc:1397] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:24.996677 19968 raft_consensus.cc:348] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:24.996856 19968 raft_consensus.cc:374] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:24.996706 19969 raft_consensus.cc:348] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:24.996891 19968 raft_consensus.cc:729] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dcbee629874a609665979a0b357bf6, State: Initialized, Role: FOLLOWER
I20260927 02:56:24.996892 19969 raft_consensus.cc:374] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:24.996928 19969 raft_consensus.cc:729] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4808cc9659fa41e5a80e390f68588027, State: Initialized, Role: FOLLOWER
I20260927 02:56:24.997007 19968 consensus_queue.cc:260] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:24.997025 19969 consensus_queue.cc:260] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:24.997274 19968 ts_tablet_manager.cc:1428] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:24.997282 19969 ts_tablet_manager.cc:1428] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:56:24.997406 19819 heartbeater.cc:503] Master 127.0.98.62:41585 was elected leader, sending a full tablet report...
I20260927 02:56:24.997438 19953 heartbeater.cc:503] Master 127.0.98.62:41585 was elected leader, sending a full tablet report...
I20260927 02:56:24.997718 19967 raft_consensus.cc:348] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:24.997886 19967 raft_consensus.cc:374] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:24.997920 19967 raft_consensus.cc:729] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7522307c282d499ba664a7dbd5552153, State: Initialized, Role: FOLLOWER
I20260927 02:56:24.998024 19967 consensus_queue.cc:260] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:24.998298 19685 heartbeater.cc:503] Master 127.0.98.62:41585 was elected leader, sending a full tablet report...
I20260927 02:56:24.998533 19967 ts_tablet_manager.cc:1428] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260927 02:56:25.066025 19820 tablet.cc:2369] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:56:25.118031 19974 raft_consensus.cc:482] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:56:25.118165 19974 raft_consensus.cc:504] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:25.118508 19974 leader_election.cc:288] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7522307c282d499ba664a7dbd5552153 (127.0.98.1:40423), 4808cc9659fa41e5a80e390f68588027 (127.0.98.2:42767)
I20260927 02:56:25.121140 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.118695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40423 (local address 127.0.98.3:34917)
0927 02:56:25.118747 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.118780 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.118825 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.118983 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.118984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.118986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:25.119047 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.119048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.119655 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.119656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.120377 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.120379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.120913 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.120914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.120981 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.120984 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.121027 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.121063 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":2}
I20260927 02:56:25.121140 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.118798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34917 (local address 127.0.98.1:40423)
0927 02:56:25.118844 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.118846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.118858 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:56:25.118887 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.118888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.118899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.118965 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:25.119112 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.119581 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.120481 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.120841 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.120899 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.120909 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.121045 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.121079 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:56:25.121228 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.118813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57887 (local address 127.0.98.2:42767)
0927 02:56:25.118866 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.118868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.118994 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:56:25.119025 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.119025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.119036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.119077 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:25.119212 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.119650 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.120565 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.120935 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.120992 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.121003 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.121131 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.121166 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:56:25.121238 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.118715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42767 (local address 127.0.98.3:57887)
0927 02:56:25.118914 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.118920 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.118961 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.119087 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.119087 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.119089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:25.119152 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.119153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.119720 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.119721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.120480 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.120481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.121007 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.121008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.121072 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.121075 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.121118 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.121155 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260927 02:56:25.121472 19639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cbc0ee179b4921a7190b7fe1db43e9" candidate_uuid: "f9dcbee629874a609665979a0b357bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7522307c282d499ba664a7dbd5552153" is_pre_election: true
I20260927 02:56:25.121546 19773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cbc0ee179b4921a7190b7fe1db43e9" candidate_uuid: "f9dcbee629874a609665979a0b357bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4808cc9659fa41e5a80e390f68588027" is_pre_election: true
I20260927 02:56:25.121613 19639 raft_consensus.cc:2454] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9dcbee629874a609665979a0b357bf6 in term 0.
I20260927 02:56:25.121694 19773 raft_consensus.cc:2454] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9dcbee629874a609665979a0b357bf6 in term 0.
I20260927 02:56:25.121838 19843 leader_election.cc:302] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7522307c282d499ba664a7dbd5552153, f9dcbee629874a609665979a0b357bf6; no voters:
I20260927 02:56:25.121979 19974 raft_consensus.cc:2781] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:56:25.122028 19974 raft_consensus.cc:482] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:56:25.122047 19974 raft_consensus.cc:3037] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:25.122715 19974 raft_consensus.cc:504] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:25.122859 19974 leader_election.cc:288] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 7522307c282d499ba664a7dbd5552153 (127.0.98.1:40423), 4808cc9659fa41e5a80e390f68588027 (127.0.98.2:42767)
I20260927 02:56:25.123008 19773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cbc0ee179b4921a7190b7fe1db43e9" candidate_uuid: "f9dcbee629874a609665979a0b357bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4808cc9659fa41e5a80e390f68588027"
I20260927 02:56:25.123014 19639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cbc0ee179b4921a7190b7fe1db43e9" candidate_uuid: "f9dcbee629874a609665979a0b357bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7522307c282d499ba664a7dbd5552153"
I20260927 02:56:25.123077 19639 raft_consensus.cc:3037] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:25.123077 19773 raft_consensus.cc:3037] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:25.123697 19639 raft_consensus.cc:2454] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9dcbee629874a609665979a0b357bf6 in term 1.
I20260927 02:56:25.123778 19773 raft_consensus.cc:2454] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9dcbee629874a609665979a0b357bf6 in term 1.
I20260927 02:56:25.123864 19843 leader_election.cc:302] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7522307c282d499ba664a7dbd5552153, f9dcbee629874a609665979a0b357bf6; no voters:
I20260927 02:56:25.123955 19974 raft_consensus.cc:2781] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:25.124104 19974 raft_consensus.cc:686] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 1 LEADER]: Becoming Leader. State: Replica: f9dcbee629874a609665979a0b357bf6, State: Running, Role: LEADER
I20260927 02:56:25.124179 19974 consensus_queue.cc:237] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } }
I20260927 02:56:25.124974 19494 catalog_manager.cc:5495] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9dcbee629874a609665979a0b357bf6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f9dcbee629874a609665979a0b357bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9dcbee629874a609665979a0b357bf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } health_report { overall_health: UNKNOWN } } }
I20260927 02:56:25.130323 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.128443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.0.98.3:42589)
0927 02:56:25.128487 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.128489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.128496 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:25.128523 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.128523 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.128536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.128592 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:25.128687 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.129105 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.129786 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.129949 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.129987 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.130095 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:25.130176 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:25.130230 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.130244 (+ 14us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.130258 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:56:25.131739 19773 raft_consensus.cc:1261] T 28cbc0ee179b4921a7190b7fe1db43e9 P 4808cc9659fa41e5a80e390f68588027 [term 1 FOLLOWER]: Refusing update from remote peer f9dcbee629874a609665979a0b357bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:25.131739 19639 raft_consensus.cc:1261] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 1 FOLLOWER]: Refusing update from remote peer f9dcbee629874a609665979a0b357bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:25.132018 19977 consensus_queue.cc:1035] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4808cc9659fa41e5a80e390f68588027" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:25.132130 19974 consensus_queue.cc:1035] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7522307c282d499ba664a7dbd5552153" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:25.134521 19980 mvcc.cc:205] Tried to move back new op lower bound from 7333797007897550848 to 7333797007869239296. Current Snapshot: MvccSnapshot[applied={T|T < 7333797007897550848}]
I20260927 02:56:25.135370 19983 mvcc.cc:205] Tried to move back new op lower bound from 7333797007897550848 to 7333797007869239296. Current Snapshot: MvccSnapshot[applied={T|T < 7333797007897550848}]
I20260927 02:56:25.136637 19493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:25.136724 19493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:25.137367 19493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e61dbe8d3ea3452eaf516f06e90be643: Sending DeleteTablet for 3 replicas of tablet 28cbc0ee179b4921a7190b7fe1db43e9
I20260927 02:56:25.137648 19887 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cbc0ee179b4921a7190b7fe1db43e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44818
I20260927 02:56:25.137691 19619 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cbc0ee179b4921a7190b7fe1db43e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47790
I20260927 02:56:25.137858 19753 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cbc0ee179b4921a7190b7fe1db43e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60590
I20260927 02:56:25.137890 19990 tablet_replica.cc:330] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153: stopping tablet replica
I20260927 02:56:25.138056 19990 raft_consensus.cc:2229] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:25.138058 19989 tablet_replica.cc:330] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: stopping tablet replica
I20260927 02:56:25.138139 19989 raft_consensus.cc:2229] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:56:25.138165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 19556
I20260927 02:56:25.138258 19990 raft_consensus.cc:2258] T 28cbc0ee179b4921a7190b7fe1db43e9 P 7522307c282d499ba664a7dbd5552153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:25.138617 19989 raft_consensus.cc:2258] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:25.139024 19989 ts_tablet_manager.cc:1905] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:25.140519 19989 ts_tablet_manager.cc:1918] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:25.140588 19989 log.cc:1196] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/28cbc0ee179b4921a7190b7fe1db43e9
I20260927 02:56:25.140857 19989 ts_tablet_manager.cc:1939] T 28cbc0ee179b4921a7190b7fe1db43e9 P f9dcbee629874a609665979a0b357bf6: Deleting consensus metadata
I20260927 02:56:25.141206 19478 catalog_manager.cc:4841] TS f9dcbee629874a609665979a0b357bf6 (127.0.98.3:42589): tablet 28cbc0ee179b4921a7190b7fe1db43e9 (table test-table [id=752875552976405280490aff86970222]) successfully deleted
W20260927 02:56:25.143267 19481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40423 (error 108)
I20260927 02:56:25.143437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 19690
I20260927 02:56:25.144117 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.143994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40423 (local address 127.0.0.1:47798)
0927 02:56:25.144037 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.144083 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:56:25.144239 19481 catalog_manager.cc:4584] TS 7522307c282d499ba664a7dbd5552153 (127.0.98.1:40423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28cbc0ee179b4921a7190b7fe1db43e9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40423: connect: Connection refused (error 111)
I20260927 02:56:25.150985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 19824
I20260927 02:56:25.151036 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.150924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42767 (local address 127.0.0.1:60596)
0927 02:56:25.150983 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.151008 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:56:25.155364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 19462
2026-09-27T02:56:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1858 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T02:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:25Z Disabled control of system clock
I20260927 02:56:25.179893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40623
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:25.255957 19996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:25.256163 19996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:25.256191 19996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:25.257309 19996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:25.257347 19996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:25.257359 19996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:25.257371 19996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:25.257395 19996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:25.258678 19996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:25.258921 19996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:25.259177 19996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:25.263454 20004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:25.263464 20002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:25.263454 20001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:25.263693 19996 server_base.cc:1034] running on GCE node
I20260927 02:56:25.263926 19996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:25.264164 19996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:25.265296 19996 hybrid_clock.cc:648] HybridClock initialized: now 1790477785265276 us; error 31 us; skew 500 ppm
I20260927 02:56:25.266371 19996 webserver.cc:460] Webserver started at http://127.0.98.62:37901/ using document root <none> and password file <none>
I20260927 02:56:25.266579 19996 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:25.266633 19996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:25.266798 19996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:25.267750 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "c19c5c43d8ca4e68bf657e4e173b416c"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "9226567b002fd0f3ab5f1de7674c668f"
server_key_iv: "4bce6a7e2e90b050b803f1413b5a1431"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.268061 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "c19c5c43d8ca4e68bf657e4e173b416c"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "9226567b002fd0f3ab5f1de7674c668f"
server_key_iv: "4bce6a7e2e90b050b803f1413b5a1431"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.269140 19996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:25.269713 20010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.269858 19996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:25.269933 19996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "c19c5c43d8ca4e68bf657e4e173b416c"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "9226567b002fd0f3ab5f1de7674c668f"
server_key_iv: "4bce6a7e2e90b050b803f1413b5a1431"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.269994 19996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:25.299499 19996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:25.299773 19996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:25.299901 19996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:25.303421 19996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40623
I20260927 02:56:25.303452 20062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40623 every 8 connection(s)
I20260927 02:56:25.303738 19996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:25.304575 20063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:25.306632 20063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c: Bootstrap starting.
I20260927 02:56:25.307168 20063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:25.307403 20063 log.cc:824] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:25.307940 20063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c: No bootstrap required, opened a new log
I20260927 02:56:25.309410 20063 raft_consensus.cc:348] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19c5c43d8ca4e68bf657e4e173b416c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } }
I20260927 02:56:25.309540 20063 raft_consensus.cc:374] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:25.309561 20063 raft_consensus.cc:729] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19c5c43d8ca4e68bf657e4e173b416c, State: Initialized, Role: FOLLOWER
I20260927 02:56:25.309659 20063 consensus_queue.cc:260] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19c5c43d8ca4e68bf657e4e173b416c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } }
I20260927 02:56:25.309706 20063 raft_consensus.cc:388] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:25.309724 20063 raft_consensus.cc:482] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:25.309751 20063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:25.310168 20063 raft_consensus.cc:504] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19c5c43d8ca4e68bf657e4e173b416c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } }
I20260927 02:56:25.310245 20063 leader_election.cc:302] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c19c5c43d8ca4e68bf657e4e173b416c; no voters:
I20260927 02:56:25.310366 20063 leader_election.cc:288] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:25.310421 20066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:25.310571 20066 raft_consensus.cc:686] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [term 1 LEADER]: Becoming Leader. State: Replica: c19c5c43d8ca4e68bf657e4e173b416c, State: Running, Role: LEADER
I20260927 02:56:25.310575 20063 sys_catalog.cc:564] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:25.310670 20066 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19c5c43d8ca4e68bf657e4e173b416c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } }
I20260927 02:56:25.311141 20067 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c19c5c43d8ca4e68bf657e4e173b416c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19c5c43d8ca4e68bf657e4e173b416c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } } }
I20260927 02:56:25.311215 20067 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:25.311405 20075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:25.311607 20068 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c19c5c43d8ca4e68bf657e4e173b416c. Latest consensus state: current_term: 1 leader_uuid: "c19c5c43d8ca4e68bf657e4e173b416c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19c5c43d8ca4e68bf657e4e173b416c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } } }
I20260927 02:56:25.311676 20068 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:25.311868 20075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:25.313225 20075 catalog_manager.cc:1347] Generated new cluster ID: 04d9401b62a34464a47d430f2e5efb9f
I20260927 02:56:25.313270 20075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:25.313423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 19996
I20260927 02:56:25.313526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:25.313699 392 external_mini_cluster.cc:1428] Setting key b80c7c512a05fad9817537cd4d664ca5
I20260927 02:56:25.317080 20075 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:25.317548 20075 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:25.319621 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.314120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.0.98.62:40623)
0927 02:56:25.314301 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.314309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.314492 (+ 183us) server_negotiation.cc:407] Connection header received
0927 02:56:25.314619 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.314625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.314687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.314847 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:25.315050 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.316419 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.317509 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.318089 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.318201 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.318868 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:25.318872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:25.318872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:25.319064 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:25.319101 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:25.319114 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:25.319170 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.319249 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.319329 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260927 02:56:25.320392 20075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c: Generated new TSK 0
I20260927 02:56:25.320544 20075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:25.324645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40623
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:56:25.402069 20087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:25.402249 20087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:25.402268 20087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:25.403422 20087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:25.403461 20087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:25.403483 20087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:56:25.404767 20087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:25.404956 20087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:25.405211 20087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:25.409755 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:25.409777 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:25.409744 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:25.410487 20087 server_base.cc:1034] running on GCE node
I20260927 02:56:25.410660 20087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:25.410907 20087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:25.412212 20087 hybrid_clock.cc:648] HybridClock initialized: now 1790477785412198 us; error 26 us; skew 500 ppm
I20260927 02:56:25.413564 20087 webserver.cc:460] Webserver started at http://127.0.98.1:45477/ using document root <none> and password file <none>
I20260927 02:56:25.413806 20087 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:25.413861 20087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:25.413995 20087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:25.415299 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "8d04a449d4e0ac797e25d7f70086cf24"
server_key_iv: "ac995ea3d06eaf0de35c6f73ba353ef8"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.415670 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "8d04a449d4e0ac797e25d7f70086cf24"
server_key_iv: "ac995ea3d06eaf0de35c6f73ba353ef8"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.417186 20087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:25.418033 20101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.418201 20087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:56:25.418300 20087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/wal
uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "8d04a449d4e0ac797e25d7f70086cf24"
server_key_iv: "ac995ea3d06eaf0de35c6f73ba353ef8"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.418373 20087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:25.427552 20087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:25.427788 20087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:25.427894 20087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:25.428082 20087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:25.428380 20087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:25.428417 20087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.428444 20087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:25.428467 20087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.434791 20087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41711
I20260927 02:56:25.434825 20214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41711 every 8 connection(s)
I20260927 02:56:25.435127 20087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:56:25.439610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 20087
I20260927 02:56:25.439709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:56:25.439858 392 external_mini_cluster.cc:1428] Setting key a72e8e63feca8653540ffddd2aace50e
I20260927 02:56:25.440150 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.435756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55749 (local address 127.0.98.62:40623)
0927 02:56:25.435821 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.435824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.436044 (+ 220us) server_negotiation.cc:407] Connection header received
0927 02:56:25.436499 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.436500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.436515 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.436581 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:25.437169 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.437930 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.438863 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.439051 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.439103 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.439873 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:25.439875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:25.439875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:25.439879 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:25.439895 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:25.439899 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:25.439944 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.440040 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.440075 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:56:25.440338 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.435760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40623 (local address 127.0.98.1:55749)
0927 02:56:25.435931 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.435962 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.436418 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.436632 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.436635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.436643 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:25.437098 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.437102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.438003 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.438003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.438752 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.438790 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.439113 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.439114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.439218 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.439729 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:25.439729 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:25.439766 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:25.439794 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:25.439952 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:25.439959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:25.439962 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.440039 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.440138 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260927 02:56:25.440691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40623
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:56:25.440969 20215 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40623
I20260927 02:56:25.441058 20215 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:25.441221 20215 heartbeater.cc:511] Master 127.0.98.62:40623 requested a full tablet report, sending...
I20260927 02:56:25.441675 20027 ts_manager.cc:194] Registered new tserver with Master: 0d9c47e6ba5c4e8d9f576eaf9793a5c6 (127.0.98.1:41711)
I20260927 02:56:25.442246 20027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55749
W20260927 02:56:25.518013 20218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:25.518193 20218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:25.518211 20218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:25.519339 20218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:25.519377 20218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:25.519404 20218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:56:25.520715 20218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:25.520927 20218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:25.521185 20218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:25.525619 20226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:25.525615 20223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:25.525663 20224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:25.525873 20218 server_base.cc:1034] running on GCE node
I20260927 02:56:25.526031 20218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:25.526202 20218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:25.527323 20218 hybrid_clock.cc:648] HybridClock initialized: now 1790477785527308 us; error 27 us; skew 500 ppm
I20260927 02:56:25.528398 20218 webserver.cc:460] Webserver started at http://127.0.98.2:40637/ using document root <none> and password file <none>
I20260927 02:56:25.528578 20218 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:25.528616 20218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:25.528702 20218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:25.529632 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/data/instance:
uuid: "9773837bc51c4a80b42b169aa6963243"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "3186c9a0c68cd15027ff05895327cf94"
server_key_iv: "98a119a4f790a358b3eaa0be09521589"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.529910 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9773837bc51c4a80b42b169aa6963243"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "3186c9a0c68cd15027ff05895327cf94"
server_key_iv: "98a119a4f790a358b3eaa0be09521589"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.531040 20218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:25.531670 20232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.531823 20218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.531909 20218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/wal
uuid: "9773837bc51c4a80b42b169aa6963243"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "3186c9a0c68cd15027ff05895327cf94"
server_key_iv: "98a119a4f790a358b3eaa0be09521589"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.531965 20218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:25.547091 20218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:25.547346 20218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:25.547470 20218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:25.547672 20218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:25.547966 20218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:25.548000 20218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.548031 20218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:25.548058 20218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.553850 20218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35061
I20260927 02:56:25.553906 20345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35061 every 8 connection(s)
I20260927 02:56:25.554203 20218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:56:25.555289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 20218
I20260927 02:56:25.555394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:56:25.555564 392 external_mini_cluster.cc:1428] Setting key 1bace38aeca6fb7a0dd52fa3790de5be
I20260927 02:56:25.556439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40623
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:56:25.559154 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.554844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46177 (local address 127.0.98.62:40623)
0927 02:56:25.554887 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.554889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.555136 (+ 247us) server_negotiation.cc:407] Connection header received
0927 02:56:25.555660 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.555661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.555673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.555724 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:25.556374 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.556928 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.557866 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.558068 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.558122 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.558915 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:25.558916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:25.558917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:25.558921 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:25.558936 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:25.558940 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:25.558982 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.559045 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.559079 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:56:25.559340 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.554831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40623 (local address 127.0.98.2:46177)
0927 02:56:25.555016 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.555045 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.555561 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.555791 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.555796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.555806 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:25.556288 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.556293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.557004 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.557006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.557782 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.557784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.558143 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.558144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.558272 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.558791 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:25.558791 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:25.558823 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:25.558844 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:25.558982 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:25.558986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:25.558989 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.559049 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.559139 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260927 02:56:25.559922 20346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40623
I20260927 02:56:25.560020 20346 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:25.560211 20346 heartbeater.cc:511] Master 127.0.98.62:40623 requested a full tablet report, sending...
I20260927 02:56:25.560592 20027 ts_manager.cc:194] Registered new tserver with Master: 9773837bc51c4a80b42b169aa6963243 (127.0.98.2:35061)
I20260927 02:56:25.561067 20027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46177
W20260927 02:56:25.635516 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:25.635732 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:25.635754 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:25.636837 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:25.636870 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:25.636895 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:56:25.638139 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:25.638324 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:25.638559 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:25.643126 20354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:25.643157 20357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:25.643118 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:25.643210 20349 server_base.cc:1034] running on GCE node
I20260927 02:56:25.643529 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:25.643752 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:25.644881 20349 hybrid_clock.cc:648] HybridClock initialized: now 1790477785644870 us; error 30 us; skew 500 ppm
I20260927 02:56:25.645969 20349 webserver.cc:460] Webserver started at http://127.0.98.3:45939/ using document root <none> and password file <none>
I20260927 02:56:25.646173 20349 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:25.646219 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:25.646341 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:25.647330 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/data/instance:
uuid: "0ec0120562d249b69981277870ebbab7"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "938bd9b6c1a2f6bc4a77d8632ac20471"
server_key_iv: "8319bddbf46fadc46d6e8c1079a606ac"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.647630 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0ec0120562d249b69981277870ebbab7"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "938bd9b6c1a2f6bc4a77d8632ac20471"
server_key_iv: "8319bddbf46fadc46d6e8c1079a606ac"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.648722 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:25.649384 20363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.649539 20349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:25.649631 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/wal
uuid: "0ec0120562d249b69981277870ebbab7"
format_stamp: "Formatted at 2026-09-27 02:56:25 on dist-test-slave-mfx9"
server_key: "938bd9b6c1a2f6bc4a77d8632ac20471"
server_key_iv: "8319bddbf46fadc46d6e8c1079a606ac"
server_key_version: "encryptionkey@0"
I20260927 02:56:25.649703 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:25.657233 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:25.657425 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:25.657508 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:25.657648 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:56:25.657904 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:56:25.657938 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.657963 20349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:56:25.657979 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:25.663051 20349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45415
I20260927 02:56:25.663089 20476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45415 every 8 connection(s)
I20260927 02:56:25.663344 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:56:25.668015 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.663931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51235 (local address 127.0.98.62:40623)
0927 02:56:25.663992 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.663994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.664176 (+ 182us) server_negotiation.cc:407] Connection header received
0927 02:56:25.664623 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.664624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.664640 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.664686 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:25.665283 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.665760 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.666687 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.667019 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.667076 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.667763 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:25.667764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:25.667764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:25.667769 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:25.667783 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:25.667787 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:25.667839 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.667911 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.667939 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:56:25.668175 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.663909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40623 (local address 127.0.98.3:51235)
0927 02:56:25.664075 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.664104 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.664535 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.664755 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.664758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.664766 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:25.665203 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.665206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.665840 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.665841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.666606 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.666608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.667090 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.667091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.667205 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.667641 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:56:25.667642 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:25.667667 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:56:25.667688 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:25.667842 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:25.667847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:25.667850 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.667909 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.668000 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260927 02:56:25.668562 20477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40623
I20260927 02:56:25.668640 20477 heartbeater.cc:464] Registering TS with master...
I20260927 02:56:25.668826 20477 heartbeater.cc:511] Master 127.0.98.62:40623 requested a full tablet report, sending...
I20260927 02:56:25.669145 20027 ts_manager.cc:194] Registered new tserver with Master: 0ec0120562d249b69981277870ebbab7 (127.0.98.3:45415)
I20260927 02:56:25.669461 20027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51235
I20260927 02:56:25.670809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 20349
I20260927 02:56:25.670912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:56:25.671036 392 external_mini_cluster.cc:1428] Setting key b9a1f39ceb88dc96605df24900e82e5b
I20260927 02:56:25.671715 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:56:25.675246 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.672889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.0.98.62:40623)
0927 02:56:25.672926 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.672928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.673032 (+ 104us) server_negotiation.cc:407] Connection header received
0927 02:56:25.673067 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.673067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.673075 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.673119 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:25.673239 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.673673 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.674372 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.674543 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.674595 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.675087 (+ 492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:25.675088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:25.675089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:25.675092 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:25.675104 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:25.675107 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:25.675145 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.675176 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.675193 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:56:25.676545 20027 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:56:25.678001 20027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:56:25.683802 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.680111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35061 (local address 127.0.0.1:43412)
0927 02:56:25.680479 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.680488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.680539 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.680795 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.680796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.680798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:25.680847 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.680849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.682017 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.682018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.682861 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.682864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.683519 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.683520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.683596 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.683610 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.683678 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.683729 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260927 02:56:25.683812 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.680226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.0.98.2:35061)
0927 02:56:25.680418 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.680421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.680599 (+ 178us) server_negotiation.cc:407] Connection header received
0927 02:56:25.680642 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.680644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.680658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.680729 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:25.680931 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.681946 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.682956 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.683436 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.683508 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.683592 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.683690 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.683729 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260927 02:56:25.684188 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.679978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.0.98.3:45415)
0927 02:56:25.680447 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.680452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.680464 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:56:25.680488 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.680490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.680503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.680570 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:25.680785 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.681502 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.682943 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.683637 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.683702 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.683781 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.684082 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.684117 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":67,"threads_started":1}
I20260927 02:56:25.684182 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.679926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45415 (local address 127.0.0.1:47300)
0927 02:56:25.680224 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.680262 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.680321 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.680619 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.680619 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.680621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:25.680652 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.680653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.681577 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.681578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.682851 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.682854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.683817 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.683818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.684014 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.684018 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.684066 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.684106 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260927 02:56:25.684559 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.680039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.0.98.1:41711)
0927 02:56:25.680210 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.680215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.680330 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:56:25.680400 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.680415 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.680432 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.680530 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:25.680664 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.681733 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.682966 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.683637 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.683702 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.683783 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.684451 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.684487 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260927 02:56:25.684512 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.679935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41711 (local address 127.0.0.1:44966)
0927 02:56:25.680236 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.680262 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.680321 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.680536 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.680539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.680541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:25.680598 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.680600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.681819 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.681820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.682889 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.682891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.683819 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.683819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.684362 (+ 543us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.684365 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.684414 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.684453 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":479,"thread_start_us":84,"threads_started":1}
I20260927 02:56:25.684499 20280 tablet_service.cc:1467] Processing CreateTablet for tablet 07ad85e6b7ae4150ab76c5c1ef12658e (DEFAULT_TABLE table=test-table [id=8c6318433f1c44e2ab0ecbf41bceb9dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:25.684824 20280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ad85e6b7ae4150ab76c5c1ef12658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:25.685189 20411 tablet_service.cc:1467] Processing CreateTablet for tablet 07ad85e6b7ae4150ab76c5c1ef12658e (DEFAULT_TABLE table=test-table [id=8c6318433f1c44e2ab0ecbf41bceb9dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:25.685287 20149 tablet_service.cc:1467] Processing CreateTablet for tablet 07ad85e6b7ae4150ab76c5c1ef12658e (DEFAULT_TABLE table=test-table [id=8c6318433f1c44e2ab0ecbf41bceb9dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:56:25.685437 20411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ad85e6b7ae4150ab76c5c1ef12658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:25.685622 20149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ad85e6b7ae4150ab76c5c1ef12658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:25.687916 20491 tablet_bootstrap.cc:492] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: Bootstrap starting.
I20260927 02:56:25.688478 20491 tablet_bootstrap.cc:654] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:25.688521 20492 tablet_bootstrap.cc:492] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: Bootstrap starting.
I20260927 02:56:25.688755 20491 log.cc:824] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:25.688913 20493 tablet_bootstrap.cc:492] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6: Bootstrap starting.
I20260927 02:56:25.689133 20492 tablet_bootstrap.cc:654] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:25.689394 20492 log.cc:824] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:25.689419 20493 tablet_bootstrap.cc:654] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:25.689666 20493 log.cc:824] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:25.690114 20491 tablet_bootstrap.cc:492] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: No bootstrap required, opened a new log
I20260927 02:56:25.690146 20492 tablet_bootstrap.cc:492] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: No bootstrap required, opened a new log
I20260927 02:56:25.690178 20491 ts_tablet_manager.cc:1397] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:25.690205 20492 ts_tablet_manager.cc:1397] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:25.690371 20493 tablet_bootstrap.cc:492] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6: No bootstrap required, opened a new log
I20260927 02:56:25.690433 20493 ts_tablet_manager.cc:1397] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:56:25.692013 20492 raft_consensus.cc:348] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.692142 20492 raft_consensus.cc:374] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:25.692176 20492 raft_consensus.cc:729] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9773837bc51c4a80b42b169aa6963243, State: Initialized, Role: FOLLOWER
I20260927 02:56:25.692286 20492 consensus_queue.cc:260] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.692548 20492 ts_tablet_manager.cc:1428] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:56:25.692566 20346 heartbeater.cc:503] Master 127.0.98.62:40623 was elected leader, sending a full tablet report...
I20260927 02:56:25.692576 20491 raft_consensus.cc:348] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.692716 20491 raft_consensus.cc:374] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:25.692749 20491 raft_consensus.cc:729] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec0120562d249b69981277870ebbab7, State: Initialized, Role: FOLLOWER
I20260927 02:56:25.692772 20493 raft_consensus.cc:348] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.692905 20493 raft_consensus.cc:374] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:25.692943 20493 raft_consensus.cc:729] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9c47e6ba5c4e8d9f576eaf9793a5c6, State: Initialized, Role: FOLLOWER
I20260927 02:56:25.693041 20493 consensus_queue.cc:260] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.693277 20493 ts_tablet_manager.cc:1428] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:56:25.693259 20491 consensus_queue.cc:260] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.693440 20215 heartbeater.cc:503] Master 127.0.98.62:40623 was elected leader, sending a full tablet report...
I20260927 02:56:25.693509 20477 heartbeater.cc:503] Master 127.0.98.62:40623 was elected leader, sending a full tablet report...
I20260927 02:56:25.693496 20491 ts_tablet_manager.cc:1428] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:56:25.710870 20497 raft_consensus.cc:482] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:56:25.710966 20497 raft_consensus.cc:504] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.711215 20497 leader_election.cc:288] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d9c47e6ba5c4e8d9f576eaf9793a5c6 (127.0.98.1:41711), 0ec0120562d249b69981277870ebbab7 (127.0.98.3:45415)
I20260927 02:56:25.713809 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.711346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45415 (local address 127.0.98.2:58895)
0927 02:56:25.711381 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.711391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.711428 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.711638 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.711639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.711641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:25.711704 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.711706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.712384 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.712385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.713097 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.713099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.713612 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.713614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.713677 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.713680 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.713718 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.713747 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:56:25.713842 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.711473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58895 (local address 127.0.98.3:45415)
0927 02:56:25.711513 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.711515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.711523 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:25.711544 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.711545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.711554 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.711600 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:25.711797 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.712307 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.713190 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.713538 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.713601 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.713610 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.713745 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.713782 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:56:25.713886 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.711383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59903 (local address 127.0.98.1:41711)
0927 02:56:25.711424 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.711426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.711557 (+ 131us) server_negotiation.cc:407] Connection header received
0927 02:56:25.711637 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.711638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.711648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.711705 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:56:25.711902 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.712311 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.713209 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.713542 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.713601 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.713610 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.713799 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.713834 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:56:25.713904 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.711336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41711 (local address 127.0.98.2:59903)
0927 02:56:25.711496 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.711504 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:25.711553 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:25.711749 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:25.711750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:25.711753 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:56:25.711829 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.711830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.712392 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.712393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.713125 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:25.713127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.713618 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.713619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:25.713722 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.713726 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:56:25.713787 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:56:25.713822 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":21,"thread_start_us":79,"threads_started":1}
I20260927 02:56:25.714087 20431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ad85e6b7ae4150ab76c5c1ef12658e" candidate_uuid: "9773837bc51c4a80b42b169aa6963243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec0120562d249b69981277870ebbab7" is_pre_election: true
I20260927 02:56:25.714139 20169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ad85e6b7ae4150ab76c5c1ef12658e" candidate_uuid: "9773837bc51c4a80b42b169aa6963243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" is_pre_election: true
I20260927 02:56:25.714213 20431 raft_consensus.cc:2454] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9773837bc51c4a80b42b169aa6963243 in term 0.
I20260927 02:56:25.714244 20169 raft_consensus.cc:2454] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9773837bc51c4a80b42b169aa6963243 in term 0.
I20260927 02:56:25.714417 20236 leader_election.cc:302] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9c47e6ba5c4e8d9f576eaf9793a5c6, 9773837bc51c4a80b42b169aa6963243; no voters:
I20260927 02:56:25.714572 20497 raft_consensus.cc:2781] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:56:25.714634 20497 raft_consensus.cc:482] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:56:25.714671 20497 raft_consensus.cc:3037] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:25.715329 20497 raft_consensus.cc:504] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.715457 20497 leader_election.cc:288] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [CANDIDATE]: Term 1 election: Requested vote from peers 0d9c47e6ba5c4e8d9f576eaf9793a5c6 (127.0.98.1:41711), 0ec0120562d249b69981277870ebbab7 (127.0.98.3:45415)
I20260927 02:56:25.715584 20431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ad85e6b7ae4150ab76c5c1ef12658e" candidate_uuid: "9773837bc51c4a80b42b169aa6963243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec0120562d249b69981277870ebbab7"
I20260927 02:56:25.715642 20431 raft_consensus.cc:3037] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:25.716300 20431 raft_consensus.cc:2454] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9773837bc51c4a80b42b169aa6963243 in term 1.
I20260927 02:56:25.716405 20169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ad85e6b7ae4150ab76c5c1ef12658e" candidate_uuid: "9773837bc51c4a80b42b169aa6963243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6"
I20260927 02:56:25.716480 20234 leader_election.cc:302] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec0120562d249b69981277870ebbab7, 9773837bc51c4a80b42b169aa6963243; no voters:
I20260927 02:56:25.716485 20169 raft_consensus.cc:3037] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:25.716645 20497 raft_consensus.cc:2781] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:25.716794 20497 raft_consensus.cc:686] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 1 LEADER]: Becoming Leader. State: Replica: 9773837bc51c4a80b42b169aa6963243, State: Running, Role: LEADER
I20260927 02:56:25.716881 20497 consensus_queue.cc:237] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } }
I20260927 02:56:25.717366 20169 raft_consensus.cc:2454] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9773837bc51c4a80b42b169aa6963243 in term 1.
I20260927 02:56:25.717618 20026 catalog_manager.cc:5495] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9773837bc51c4a80b42b169aa6963243 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9773837bc51c4a80b42b169aa6963243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9773837bc51c4a80b42b169aa6963243" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35061 } health_report { overall_health: HEALTHY } } }
I20260927 02:56:25.729812 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.727897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.0.98.2:35061)
0927 02:56:25.727950 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:25.727951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:25.727958 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:25.728012 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:25.728013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:25.728026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:25.728075 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:25.728157 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.728584 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.729259 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:25.729429 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:25.729469 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:25.729568 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:25.729648 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:25.729698 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:25.729713 (+ 15us) server_negotiation.cc:299] Negotiation successful
0927 02:56:25.729727 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:56:25.731293 20169 raft_consensus.cc:1261] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0d9c47e6ba5c4e8d9f576eaf9793a5c6 [term 1 FOLLOWER]: Refusing update from remote peer 9773837bc51c4a80b42b169aa6963243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:25.731295 20431 raft_consensus.cc:1261] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 1 FOLLOWER]: Refusing update from remote peer 9773837bc51c4a80b42b169aa6963243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:25.731580 20501 consensus_queue.cc:1035] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec0120562d249b69981277870ebbab7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:25.731673 20497 consensus_queue.cc:1035] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9c47e6ba5c4e8d9f576eaf9793a5c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:25.733798 20504 mvcc.cc:205] Tried to move back new op lower bound from 7333797010353283072 to 7333797010296987648. Current Snapshot: MvccSnapshot[applied={T|T < 7333797010353283072}]
I20260927 02:56:25.734594 20505 mvcc.cc:205] Tried to move back new op lower bound from 7333797010353283072 to 7333797010296987648. Current Snapshot: MvccSnapshot[applied={T|T < 7333797010353283072}]
I20260927 02:56:25.735862 20026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:25.735934 20026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:56:25.736564 20026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c19c5c43d8ca4e68bf657e4e173b416c: Sending DeleteTablet for 3 replicas of tablet 07ad85e6b7ae4150ab76c5c1ef12658e
I20260927 02:56:25.736866 20149 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ad85e6b7ae4150ab76c5c1ef12658e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:25 UTC) from {username='slave'} at 127.0.0.1:44966
I20260927 02:56:25.736866 20411 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ad85e6b7ae4150ab76c5c1ef12658e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:25 UTC) from {username='slave'} at 127.0.0.1:47300
I20260927 02:56:25.736883 20280 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ad85e6b7ae4150ab76c5c1ef12658e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:56:25 UTC) from {username='slave'} at 127.0.0.1:43412
I20260927 02:56:25.737068 20515 tablet_replica.cc:330] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: stopping tablet replica
I20260927 02:56:25.737156 20515 raft_consensus.cc:2229] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:56:25.737248 20515 raft_consensus.cc:2258] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:25.737327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 20087
I20260927 02:56:25.739866 20515 ts_tablet_manager.cc:1905] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:25.741194 20515 ts_tablet_manager.cc:1918] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:25.741247 20515 log.cc:1196] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-1/wal/wals/07ad85e6b7ae4150ab76c5c1ef12658e
I20260927 02:56:25.741441 20515 ts_tablet_manager.cc:1939] T 07ad85e6b7ae4150ab76c5c1ef12658e P 9773837bc51c4a80b42b169aa6963243: Deleting consensus metadata
I20260927 02:56:25.741745 20012 catalog_manager.cc:4841] TS 9773837bc51c4a80b42b169aa6963243 (127.0.98.2:35061): tablet 07ad85e6b7ae4150ab76c5c1ef12658e (table test-table [id=8c6318433f1c44e2ab0ecbf41bceb9dc]) successfully deleted
I20260927 02:56:25.746855 20516 tablet_replica.cc:330] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: stopping tablet replica
I20260927 02:56:25.746938 20516 raft_consensus.cc:2229] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:56:25.747037 20516 raft_consensus.cc:2258] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:56:25.747383 20516 ts_tablet_manager.cc:1905] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:56:25.748714 20516 ts_tablet_manager.cc:1918] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:56:25.748773 20516 log.cc:1196] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: Deleting WAL directory at /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477553417353-392-0/minicluster-data/ts-2/wal/wals/07ad85e6b7ae4150ab76c5c1ef12658e
I20260927 02:56:25.748973 20516 ts_tablet_manager.cc:1939] T 07ad85e6b7ae4150ab76c5c1ef12658e P 0ec0120562d249b69981277870ebbab7: Deleting consensus metadata
I20260927 02:56:25.749311 20012 catalog_manager.cc:4841] TS 0ec0120562d249b69981277870ebbab7 (127.0.98.3:45415): tablet 07ad85e6b7ae4150ab76c5c1ef12658e (table test-table [id=8c6318433f1c44e2ab0ecbf41bceb9dc]) successfully deleted
W20260927 02:56:25.750267 20014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41711 (error 108)
I20260927 02:56:25.750447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 20218
I20260927 02:56:25.751731 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:25.751597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41711 (local address 127.0.0.1:44978)
0927 02:56:25.751654 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:25.751699 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 02:56:25.751827 20014 catalog_manager.cc:4584] TS 0d9c47e6ba5c4e8d9f576eaf9793a5c6 (127.0.98.1:41711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07ad85e6b7ae4150ab76c5c1ef12658e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41711: connect: Connection refused (error 111)
I20260927 02:56:25.754428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 20349
I20260927 02:56:25.758167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 19996
2026-09-27T02:56:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (602 ms)
[----------] 6 tests from EncryptionPolicyTest (7532 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:25 dist-test-slave-mfx9 krb5kdc[20520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:25 dist-test-slave-mfx9 krb5kdc[20520](info): set up 2 sockets
Sep 27 02:56:25 dist-test-slave-mfx9 krb5kdc[20520](info): commencing operation
krb5kdc: starting...
W20260927 02:56:26.698364 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:26 dist-test-slave-mfx9 krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:26.802562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:26.878654 20533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:26.878970 20533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:26.878994 20533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:26.880208 20533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:26.880247 20533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:26.880259 20533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:26.880270 20533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:26.880280 20533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:26.881695 20533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:26.881865 20533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:26.882023 20533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:26.886459 20539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:26.886444 20538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:26.886438 20541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:26.886634 20533 server_base.cc:1034] running on GCE node
I20260927 02:56:26.886920 20533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:26.887145 20533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:26.888296 20533 hybrid_clock.cc:648] HybridClock initialized: now 1790477786888284 us; error 29 us; skew 500 ppm
Sep 27 02:56:26 dist-test-slave-mfx9 krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:26.890523 20533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:26.890972 20533 webserver.cc:460] Webserver started at http://127.0.0.1:41707/ using document root <none> and password file <none>
I20260927 02:56:26.891152 20533 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:26.891213 20533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:26.891335 20533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:26.892304 20533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "13312328b5a1463ca038729f764658d1"
format_stamp: "Formatted at 2026-09-27 02:56:26 on dist-test-slave-mfx9"
server_key: "7afde8a17dc11c824980bbb404c0962b"
server_key_iv: "6480cbdfdfbff11f6d0c8cb7af9d392b"
server_key_version: "encryptionkey@0"
I20260927 02:56:26.892608 20533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "13312328b5a1463ca038729f764658d1"
format_stamp: "Formatted at 2026-09-27 02:56:26 on dist-test-slave-mfx9"
server_key: "7afde8a17dc11c824980bbb404c0962b"
server_key_iv: "6480cbdfdfbff11f6d0c8cb7af9d392b"
server_key_version: "encryptionkey@0"
I20260927 02:56:26.893743 20533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:26.894347 20548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:26.894479 20533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:26.894553 20533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "13312328b5a1463ca038729f764658d1"
format_stamp: "Formatted at 2026-09-27 02:56:26 on dist-test-slave-mfx9"
server_key: "7afde8a17dc11c824980bbb404c0962b"
server_key_iv: "6480cbdfdfbff11f6d0c8cb7af9d392b"
server_key_version: "encryptionkey@0"
I20260927 02:56:26.894615 20533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:26.909137 20533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:26.910670 20533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:26.910848 20533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:26.914307 20533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46727
I20260927 02:56:26.914346 20600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46727 every 8 connection(s)
I20260927 02:56:26.914608 20533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:26.915355 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:26.916103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 20533
I20260927 02:56:26.916215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:26.916390 392 external_mini_cluster.cc:1428] Setting key 50d7c28b57eb36a863aa919e2eeabc01
I20260927 02:56:26.918035 20601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1: Bootstrap starting.
I20260927 02:56:26.918534 20601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:26.918812 20601 log.cc:824] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:26.919430 20601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1: No bootstrap required, opened a new log
I20260927 02:56:26.921597 20601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13312328b5a1463ca038729f764658d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260927 02:56:26.921761 20601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:26.921794 20601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13312328b5a1463ca038729f764658d1, State: Initialized, Role: FOLLOWER
I20260927 02:56:26.921911 20601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13312328b5a1463ca038729f764658d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260927 02:56:26.921984 20601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:26.922020 20601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:26.922066 20601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:56:26 dist-test-slave-mfx9 krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:56:26.922531 20601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13312328b5a1463ca038729f764658d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260927 02:56:26.922612 20601 leader_election.cc:302] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13312328b5a1463ca038729f764658d1; no voters:
I20260927 02:56:26.922787 20601 leader_election.cc:288] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:26.922824 20606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:26.922967 20606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [term 1 LEADER]: Becoming Leader. State: Replica: 13312328b5a1463ca038729f764658d1, State: Running, Role: LEADER
I20260927 02:56:26.923022 20601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:26.923344 20606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13312328b5a1463ca038729f764658d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } }
I20260927 02:56:26.923934 20607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13312328b5a1463ca038729f764658d1. Latest consensus state: current_term: 1 leader_uuid: "13312328b5a1463ca038729f764658d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13312328b5a1463ca038729f764658d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } } }
I20260927 02:56:26.924029 20607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:26.923918 20608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13312328b5a1463ca038729f764658d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13312328b5a1463ca038729f764658d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46727 } } }
I20260927 02:56:26.924198 20608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:26.924226 20618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:26.924746 20618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:26.926481 20618 catalog_manager.cc:1347] Generated new cluster ID: e75ac2feabb14902be8c8e29ba63cc52
I20260927 02:56:26.926555 20618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:26.926617 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:26.916864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34287 (local address 127.0.0.1:46727)
0927 02:56:26.917026 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:26.917033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:26.917055 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:56:26.917435 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:26.917440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:26.917481 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:26.917593 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:26.918207 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:26.919171 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:26.920213 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:26.920609 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:26.920679 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:26.923074 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:26.923078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:26.923078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:26.923089 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:26.924958 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:26.925371 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:26.925374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:26.925374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:26.925468 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:26.925720 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:26.925721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:26.925721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:26.925942 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:26.926042 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:26.926301 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 02:56:26.926404 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260927 02:56:26.927451 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:26.940143 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:26.928675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58123 (local address 127.0.0.1:46727)
0927 02:56:26.928711 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:26.928713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:26.934910 (+ 6197us) server_negotiation.cc:407] Connection header received
0927 02:56:26.934970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:26.934970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:26.934980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:26.935027 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:26.935496 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:26.935962 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:26.936609 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:26.936814 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:26.936872 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:26.937913 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:26.937914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:26.937915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:26.937921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:26.939147 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:26.939476 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:26.939477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:26.939477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:26.939526 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:26.939703 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:26.939704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:26.939704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:26.939801 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:26.939907 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:26.940006 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 02:56:26.940060 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:26.942580 20618 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:26.943269 20618 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:26.951148 20618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13312328b5a1463ca038729f764658d1: Generated new TSK 0
I20260927 02:56:26.951318 20618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:26.954129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 20533
W20260927 02:56:26.958015 20518 connection.cc:535] client connection to 127.0.0.1:46727 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46727): Connection reset by peer (error 104)
2026-09-27T02:56:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:27 dist-test-slave-mfx9 krb5kdc[20632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:27 dist-test-slave-mfx9 krb5kdc[20632](info): set up 2 sockets
Sep 27 02:56:27 dist-test-slave-mfx9 krb5kdc[20632](info): commencing operation
krb5kdc: starting...
W20260927 02:56:27.894366 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:27 dist-test-slave-mfx9 krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:28.002750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:28.080411 20645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:28.080606 20645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:28.080636 20645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:28.081753 20645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:28.081791 20645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:28.081809 20645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:28.081825 20645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:28.081840 20645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:28.083287 20645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:28.083514 20645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:28.083705 20645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:28.088209 20653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:28.088209 20650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:28.088303 20651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:28.088367 20645 server_base.cc:1034] running on GCE node
I20260927 02:56:28.088696 20645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:28.088934 20645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:28.090063 20645 hybrid_clock.cc:648] HybridClock initialized: now 1790477788090050 us; error 28 us; skew 500 ppm
Sep 27 02:56:28 dist-test-slave-mfx9 krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:28.092392 20645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:28.092808 20645 webserver.cc:460] Webserver started at http://127.0.0.1:43949/ using document root <none> and password file <none>
I20260927 02:56:28.092995 20645 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:28.093034 20645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:28.093118 20645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:28.094076 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "266d43dd1ee944e6bb114535f8acb0ff"
format_stamp: "Formatted at 2026-09-27 02:56:28 on dist-test-slave-mfx9"
server_key: "e92572f464f29614da26daa76e9d739c"
server_key_iv: "028fbf244f1d2a0b7b98dbe476b405a7"
server_key_version: "encryptionkey@0"
I20260927 02:56:28.094358 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "266d43dd1ee944e6bb114535f8acb0ff"
format_stamp: "Formatted at 2026-09-27 02:56:28 on dist-test-slave-mfx9"
server_key: "e92572f464f29614da26daa76e9d739c"
server_key_iv: "028fbf244f1d2a0b7b98dbe476b405a7"
server_key_version: "encryptionkey@0"
I20260927 02:56:28.095552 20645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:28.096206 20660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:28.096329 20645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:28.096401 20645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "266d43dd1ee944e6bb114535f8acb0ff"
format_stamp: "Formatted at 2026-09-27 02:56:28 on dist-test-slave-mfx9"
server_key: "e92572f464f29614da26daa76e9d739c"
server_key_iv: "028fbf244f1d2a0b7b98dbe476b405a7"
server_key_version: "encryptionkey@0"
I20260927 02:56:28.096464 20645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:28.112130 20645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:28.113785 20645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:28.113888 20645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:28.117548 20645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44145
I20260927 02:56:28.117581 20712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44145 every 8 connection(s)
I20260927 02:56:28.117901 20645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:28.118743 20713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:28.120970 20713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff: Bootstrap starting.
I20260927 02:56:28.121464 20713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:28.121687 20713 log.cc:824] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:28.122287 20713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff: No bootstrap required, opened a new log
I20260927 02:56:28.123888 20713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266d43dd1ee944e6bb114535f8acb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } }
I20260927 02:56:28.124019 20713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:28.124039 20713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266d43dd1ee944e6bb114535f8acb0ff, State: Initialized, Role: FOLLOWER
I20260927 02:56:28.124135 20713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266d43dd1ee944e6bb114535f8acb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } }
I20260927 02:56:28.124193 20713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:28.124213 20713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:28.124248 20713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:28.124679 20713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266d43dd1ee944e6bb114535f8acb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } }
I20260927 02:56:28.124758 20713 leader_election.cc:302] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 266d43dd1ee944e6bb114535f8acb0ff; no voters:
I20260927 02:56:28.124869 20713 leader_election.cc:288] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:28.124930 20716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:28.125075 20713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:28.125077 20716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [term 1 LEADER]: Becoming Leader. State: Replica: 266d43dd1ee944e6bb114535f8acb0ff, State: Running, Role: LEADER
I20260927 02:56:28.125247 20716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266d43dd1ee944e6bb114535f8acb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } }
I20260927 02:56:28.125589 20718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 266d43dd1ee944e6bb114535f8acb0ff. Latest consensus state: current_term: 1 leader_uuid: "266d43dd1ee944e6bb114535f8acb0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266d43dd1ee944e6bb114535f8acb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } } }
I20260927 02:56:28.125705 20718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:28.125614 20717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "266d43dd1ee944e6bb114535f8acb0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266d43dd1ee944e6bb114535f8acb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } } }
I20260927 02:56:28.125898 20720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:28.125909 20717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:28.126484 20720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:28.127007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 20645
I20260927 02:56:28.127103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:28.127260 392 external_mini_cluster.cc:1428] Setting key c30f58de4ed8bc3ef00cf08d44b759b6
I20260927 02:56:28.128253 20720 catalog_manager.cc:1347] Generated new cluster ID: 418035e7279849459e610c0f0dff7f46
I20260927 02:56:28.128302 20720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:28.134832 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:28.127650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57367 (local address 127.0.0.1:44145)
0927 02:56:28.130877 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:28.130884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:28.130911 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:56:28.130998 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:28.131003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:28.131043 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:28.131187 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:28.131389 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:28.132343 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:28.133113 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:28.133535 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:28.133626 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:28.134265 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:28.134268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:28.134268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:28.134422 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:28.134451 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:28.134462 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:28.134527 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:28.134616 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:56:28.134644 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3171,"thread_start_us":73,"threads_started":1}
I20260927 02:56:28.135648 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:28.145319 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:28.139064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47721 (local address 127.0.0.1:44145)
0927 02:56:28.139099 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:28.139101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:28.142893 (+ 3792us) server_negotiation.cc:407] Connection header received
0927 02:56:28.142948 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:28.142949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:28.142970 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:28.143021 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:28.143124 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:28.143579 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:28.144230 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:28.144443 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:28.144497 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:28.145107 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:28.145108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:28.145108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:28.145112 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:28.145125 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:28.145129 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:28.145173 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:28.145237 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:56:28.145249 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:28.149156 20720 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:28.149818 20720 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:28.155452 20720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 266d43dd1ee944e6bb114535f8acb0ff: Generated new TSK 0
I20260927 02:56:28.155623 20720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:28.164680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 20645
W20260927 02:56:28.168365 20630 connection.cc:535] client connection to 127.0.0.1:44145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44145): Connection reset by peer (error 104)
2026-09-27T02:56:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:28 dist-test-slave-mfx9 krb5kdc[20744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:28 dist-test-slave-mfx9 krb5kdc[20744](info): set up 2 sockets
Sep 27 02:56:28 dist-test-slave-mfx9 krb5kdc[20744](info): commencing operation
krb5kdc: starting...
W20260927 02:56:29.110432 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:29 dist-test-slave-mfx9 krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:29.215381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:29.291286 20757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:29.291471 20757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:29.291500 20757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:29.292593 20757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:29.292630 20757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:29.292649 20757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:29.292666 20757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:29.292681 20757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:29.294068 20757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:29.294286 20757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:29.294466 20757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:56:29.298916 20757 server_base.cc:1034] running on GCE node
W20260927 02:56:29.298833 20762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:29.298834 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:29.298882 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:29.299269 20757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:29.299530 20757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:29.300660 20757 hybrid_clock.cc:648] HybridClock initialized: now 1790477789300645 us; error 31 us; skew 500 ppm
Sep 27 02:56:29 dist-test-slave-mfx9 krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:29.302870 20757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:29.303260 20757 webserver.cc:460] Webserver started at http://127.0.0.1:46005/ using document root <none> and password file <none>
I20260927 02:56:29.303457 20757 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:29.303498 20757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:29.303615 20757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:29.304551 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "24c3c669e2d44930b1c8666ba46de815"
format_stamp: "Formatted at 2026-09-27 02:56:29 on dist-test-slave-mfx9"
server_key: "1453d10f556e4bc3b99911b4b3452723"
server_key_iv: "b8f27ee5021373623e673ae2291e811b"
server_key_version: "encryptionkey@0"
I20260927 02:56:29.304854 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "24c3c669e2d44930b1c8666ba46de815"
format_stamp: "Formatted at 2026-09-27 02:56:29 on dist-test-slave-mfx9"
server_key: "1453d10f556e4bc3b99911b4b3452723"
server_key_iv: "b8f27ee5021373623e673ae2291e811b"
server_key_version: "encryptionkey@0"
I20260927 02:56:29.305980 20757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:29.306560 20772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:29.306699 20757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:29.306797 20757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "24c3c669e2d44930b1c8666ba46de815"
format_stamp: "Formatted at 2026-09-27 02:56:29 on dist-test-slave-mfx9"
server_key: "1453d10f556e4bc3b99911b4b3452723"
server_key_iv: "b8f27ee5021373623e673ae2291e811b"
server_key_version: "encryptionkey@0"
I20260927 02:56:29.306862 20757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:29.309862 20757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:29.311503 20757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:29.311597 20757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:29.314956 20757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40823
I20260927 02:56:29.314982 20824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40823 every 8 connection(s)
I20260927 02:56:29.315239 20757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:29.316095 20825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:29.318039 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815: Bootstrap starting.
I20260927 02:56:29.318523 20825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:29.318732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 20757
I20260927 02:56:29.318781 20825 log.cc:824] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:29.318867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:29.319029 392 external_mini_cluster.cc:1428] Setting key 3e79fb257f4461e993b33b9e996f0d09
I20260927 02:56:29.319454 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815: No bootstrap required, opened a new log
I20260927 02:56:29.321416 20825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c669e2d44930b1c8666ba46de815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40823 } }
I20260927 02:56:29.321564 20825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:29.321602 20825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c3c669e2d44930b1c8666ba46de815, State: Initialized, Role: FOLLOWER
I20260927 02:56:29.321715 20825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c669e2d44930b1c8666ba46de815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40823 } }
I20260927 02:56:29.321791 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:29.319454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60723 (local address 127.0.0.1:40823)
0927 02:56:29.319609 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:29.319618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:29.319649 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:56:29.320596 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:29.320605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:29.320637 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:29.320773 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:29.321259 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:29.321263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:29.321263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:29.321267 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:29.321312 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:29.321325 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:29.321397 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:29.321433 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:56:29.321468 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260927 02:56:29.321811 20825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:29.321841 20825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:29.321878 20825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:29.322444 20825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c669e2d44930b1c8666ba46de815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40823 } }
I20260927 02:56:29.322542 20825 leader_election.cc:302] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24c3c669e2d44930b1c8666ba46de815; no voters:
I20260927 02:56:29.322582 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:29.322690 20825 leader_election.cc:288] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:29.322715 20830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:29.322854 20830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [term 1 LEADER]: Becoming Leader. State: Replica: 24c3c669e2d44930b1c8666ba46de815, State: Running, Role: LEADER
I20260927 02:56:29.322942 20825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:29.322993 20830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c669e2d44930b1c8666ba46de815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40823 } }
I20260927 02:56:29.323303 20830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c3c669e2d44930b1c8666ba46de815. Latest consensus state: current_term: 1 leader_uuid: "24c3c669e2d44930b1c8666ba46de815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c669e2d44930b1c8666ba46de815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40823 } } }
I20260927 02:56:29.323385 20830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:29.323308 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24c3c669e2d44930b1c8666ba46de815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c669e2d44930b1c8666ba46de815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40823 } } }
I20260927 02:56:29.323522 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815 [sys.catalog]: This master's current role is: LEADER
W20260927 02:56:29.324532 20849 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:29.324625 20849 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:29.324712 20837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:29.324921 20837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:29.326176 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:29.325333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53487 (local address 127.0.0.1:40823)
0927 02:56:29.325366 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:29.325368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:29.325374 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:56:29.325392 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:29.325393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:29.325396 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:29.325444 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:29.326001 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:29.326002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:29.326002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:29.326006 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:29.326017 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:29.326020 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:29.326064 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:29.326111 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:56:29.326122 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:56:29.326287 20837 catalog_manager.cc:1347] Generated new cluster ID: 0a7cf2c03dbc48f69cdb44c151ada7bd
I20260927 02:56:29.326332 20837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:29.351944 20837 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:29.352149 20837 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:29.364707 20837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24c3c669e2d44930b1c8666ba46de815: Generated new TSK 0
I20260927 02:56:29.364909 20837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:29.367324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 20757
2026-09-27T02:56:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:29 dist-test-slave-mfx9 krb5kdc[20856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:29 dist-test-slave-mfx9 krb5kdc[20856](info): set up 2 sockets
Sep 27 02:56:29 dist-test-slave-mfx9 krb5kdc[20856](info): commencing operation
krb5kdc: starting...
W20260927 02:56:30.298445 392 mini_kdc.cc:121] Time spent starting KDC: real 0.914s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:30 dist-test-slave-mfx9 krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:30.403247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:30.482213 20869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:30.482416 20869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:30.482438 20869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:30.483574 20869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:30.483613 20869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:30.483625 20869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:30.483636 20869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:30.483646 20869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:30.485074 20869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:30.485265 20869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:30.485428 20869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:30.489753 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:30.489792 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:30.489809 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:30.489882 20869 server_base.cc:1034] running on GCE node
I20260927 02:56:30.490156 20869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:30.490450 20869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:30.491586 20869 hybrid_clock.cc:648] HybridClock initialized: now 1790477790491567 us; error 34 us; skew 500 ppm
Sep 27 02:56:30 dist-test-slave-mfx9 krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:30.493851 20869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:30.494237 20869 webserver.cc:460] Webserver started at http://127.0.0.1:37817/ using document root <none> and password file <none>
I20260927 02:56:30.494439 20869 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:30.494484 20869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:30.494602 20869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:30.495615 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "e6e4519ce8f54fbe99dac9e39dc52d22"
format_stamp: "Formatted at 2026-09-27 02:56:30 on dist-test-slave-mfx9"
server_key: "f8eaf56a33256e6d3e95d63345d5c354"
server_key_iv: "8ad263a973c6892e80990cb6da548278"
server_key_version: "encryptionkey@0"
I20260927 02:56:30.495931 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "e6e4519ce8f54fbe99dac9e39dc52d22"
format_stamp: "Formatted at 2026-09-27 02:56:30 on dist-test-slave-mfx9"
server_key: "f8eaf56a33256e6d3e95d63345d5c354"
server_key_iv: "8ad263a973c6892e80990cb6da548278"
server_key_version: "encryptionkey@0"
I20260927 02:56:30.497119 20869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:30.497713 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:30.497889 20869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:30.497967 20869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "e6e4519ce8f54fbe99dac9e39dc52d22"
format_stamp: "Formatted at 2026-09-27 02:56:30 on dist-test-slave-mfx9"
server_key: "f8eaf56a33256e6d3e95d63345d5c354"
server_key_iv: "8ad263a973c6892e80990cb6da548278"
server_key_version: "encryptionkey@0"
I20260927 02:56:30.498018 20869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:30.508057 20869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:30.509598 20869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:30.509694 20869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:30.513155 20869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43925
I20260927 02:56:30.513200 20936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43925 every 8 connection(s)
I20260927 02:56:30.513479 20869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:30.514241 20937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:30.516348 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22: Bootstrap starting.
I20260927 02:56:30.516894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 20869
I20260927 02:56:30.516921 20937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:30.517016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:30.517194 392 external_mini_cluster.cc:1428] Setting key d2c0df40190f444714bffc196fffe97e
I20260927 02:56:30.517198 20937 log.cc:824] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:30.517889 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22: No bootstrap required, opened a new log
I20260927 02:56:30.519853 20937 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4519ce8f54fbe99dac9e39dc52d22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43925 } }
I20260927 02:56:30.519977 20937 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:30.520001 20937 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e4519ce8f54fbe99dac9e39dc52d22, State: Initialized, Role: FOLLOWER
I20260927 02:56:30.520112 20937 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4519ce8f54fbe99dac9e39dc52d22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43925 } }
I20260927 02:56:30.520185 20937 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:30.520221 20937 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:30.520264 20937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:30.520897 20937 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4519ce8f54fbe99dac9e39dc52d22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43925 } }
I20260927 02:56:30.521001 20937 leader_election.cc:302] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6e4519ce8f54fbe99dac9e39dc52d22; no voters:
I20260927 02:56:30.521152 20937 leader_election.cc:288] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:30.521191 20942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:30.521339 20942 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [term 1 LEADER]: Becoming Leader. State: Replica: e6e4519ce8f54fbe99dac9e39dc52d22, State: Running, Role: LEADER
I20260927 02:56:30.521394 20937 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:30.521425 20942 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4519ce8f54fbe99dac9e39dc52d22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43925 } }
I20260927 02:56:30.521723 20942 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e4519ce8f54fbe99dac9e39dc52d22. Latest consensus state: current_term: 1 leader_uuid: "e6e4519ce8f54fbe99dac9e39dc52d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4519ce8f54fbe99dac9e39dc52d22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43925 } } }
I20260927 02:56:30.521790 20942 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:30.522070 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6e4519ce8f54fbe99dac9e39dc52d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4519ce8f54fbe99dac9e39dc52d22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43925 } } }
I20260927 02:56:30.522490 20948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:30.522976 20948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:30.522217 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22 [sys.catalog]: This master's current role is: LEADER
Sep 27 02:56:30 dist-test-slave-mfx9 krb5kdc[20856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:56:30.524456 20948 catalog_manager.cc:1347] Generated new cluster ID: a030b0cd47f749f3ad58de4c5b3b161a
I20260927 02:56:30.524500 20948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:30.527663 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:30.517643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58615 (local address 127.0.0.1:43925)
0927 02:56:30.517799 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:30.517809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:30.517843 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:56:30.518271 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:30.518278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:30.518330 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:30.518449 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:30.519096 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:30.519953 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:30.520996 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:30.521605 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:30.521709 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:30.524325 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:30.524329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:30.524329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:30.524340 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:30.526165 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:30.526509 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:30.526511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:30.526512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:30.526600 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:30.526790 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:30.526790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:30.526791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:30.527057 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:30.527199 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:30.527343 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 02:56:30.527469 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260927 02:56:30.528607 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:30.536355 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:30.529791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51653 (local address 127.0.0.1:43925)
0927 02:56:30.529828 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:30.529829 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:30.529918 (+ 89us) server_negotiation.cc:407] Connection header received
0927 02:56:30.529978 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:30.529979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:30.529987 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:30.530029 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:30.530602 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:30.531355 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:30.532330 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:30.532623 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:30.532673 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:30.533938 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:30.533939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:30.533939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:30.533945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:30.535363 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:30.535715 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:30.535715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:30.535716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:30.535762 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:30.535926 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:30.535927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:30.535927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:30.536025 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:30.536102 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:30.536206 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 02:56:30.536269 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:30.546573 20948 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:30.547083 20948 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:30.562361 20948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6e4519ce8f54fbe99dac9e39dc52d22: Generated new TSK 0
I20260927 02:56:30.562544 20948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:30.570006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 20869
2026-09-27T02:56:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:30 dist-test-slave-mfx9 krb5kdc[20968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:30 dist-test-slave-mfx9 krb5kdc[20968](info): set up 2 sockets
Sep 27 02:56:30 dist-test-slave-mfx9 krb5kdc[20968](info): commencing operation
krb5kdc: starting...
W20260927 02:56:31.510423 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:31 dist-test-slave-mfx9 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:31.619393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:31.695151 20981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:31.695336 20981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:31.695366 20981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:31.696461 20981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:31.696513 20981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:31.696532 20981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:31.696548 20981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:31.696563 20981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:31.697986 20981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:31.698207 20981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:31.698395 20981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:31.702746 20987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:31.702873 20986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:31.702965 20981 server_base.cc:1034] running on GCE node
W20260927 02:56:31.702984 20989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:31.703230 20981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:31.703491 20981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:31.704636 20981 hybrid_clock.cc:648] HybridClock initialized: now 1790477791704619 us; error 32 us; skew 500 ppm
Sep 27 02:56:31 dist-test-slave-mfx9 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:31.706964 20981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:31.707383 20981 webserver.cc:460] Webserver started at http://127.0.0.1:43825/ using document root <none> and password file <none>
I20260927 02:56:31.707571 20981 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:31.707616 20981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:31.707736 20981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:31.708712 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "ff7c624ef4b7481a8509e8b0b9bb6519"
format_stamp: "Formatted at 2026-09-27 02:56:31 on dist-test-slave-mfx9"
server_key: "b151cf38c7b5b52aba2c01ce41401987"
server_key_iv: "bc1ade554cce322ff6dd2f3682cfc84b"
server_key_version: "encryptionkey@0"
I20260927 02:56:31.709024 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff7c624ef4b7481a8509e8b0b9bb6519"
format_stamp: "Formatted at 2026-09-27 02:56:31 on dist-test-slave-mfx9"
server_key: "b151cf38c7b5b52aba2c01ce41401987"
server_key_iv: "bc1ade554cce322ff6dd2f3682cfc84b"
server_key_version: "encryptionkey@0"
I20260927 02:56:31.710216 20981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:56:31.710914 20996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:31.711059 20981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:31.711136 20981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "ff7c624ef4b7481a8509e8b0b9bb6519"
format_stamp: "Formatted at 2026-09-27 02:56:31 on dist-test-slave-mfx9"
server_key: "b151cf38c7b5b52aba2c01ce41401987"
server_key_iv: "bc1ade554cce322ff6dd2f3682cfc84b"
server_key_version: "encryptionkey@0"
I20260927 02:56:31.711218 20981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:31.728003 20981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:31.729914 20981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:31.730046 20981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:31.733832 20981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35625
I20260927 02:56:31.733865 21048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35625 every 8 connection(s)
I20260927 02:56:31.734176 20981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:31.734988 21049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:31.737095 21049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519: Bootstrap starting.
I20260927 02:56:31.737586 21049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:31.737807 21049 log.cc:824] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:31.738392 21049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519: No bootstrap required, opened a new log
I20260927 02:56:31.739924 21049 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7c624ef4b7481a8509e8b0b9bb6519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } }
I20260927 02:56:31.740053 21049 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:31.740074 21049 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7c624ef4b7481a8509e8b0b9bb6519, State: Initialized, Role: FOLLOWER
I20260927 02:56:31.740180 21049 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7c624ef4b7481a8509e8b0b9bb6519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } }
I20260927 02:56:31.740252 21049 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:31.740278 21049 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:31.740312 21049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:31.740756 21049 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7c624ef4b7481a8509e8b0b9bb6519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } }
I20260927 02:56:31.740844 21049 leader_election.cc:302] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff7c624ef4b7481a8509e8b0b9bb6519; no voters:
I20260927 02:56:31.740954 21049 leader_election.cc:288] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:31.741003 21052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:31.741153 21049 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:31.741175 21052 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [term 1 LEADER]: Becoming Leader. State: Replica: ff7c624ef4b7481a8509e8b0b9bb6519, State: Running, Role: LEADER
I20260927 02:56:31.741304 21052 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7c624ef4b7481a8509e8b0b9bb6519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } }
I20260927 02:56:31.741858 21054 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff7c624ef4b7481a8509e8b0b9bb6519. Latest consensus state: current_term: 1 leader_uuid: "ff7c624ef4b7481a8509e8b0b9bb6519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7c624ef4b7481a8509e8b0b9bb6519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } } }
I20260927 02:56:31.741870 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff7c624ef4b7481a8509e8b0b9bb6519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7c624ef4b7481a8509e8b0b9bb6519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35625 } } }
I20260927 02:56:31.741945 21054 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:31.741961 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:31.742187 21061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:31.742740 21061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:31.743186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 20981
I20260927 02:56:31.743295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:31.743472 392 external_mini_cluster.cc:1428] Setting key 9b7be512ed9f9f0090062be46b6a33ad
I20260927 02:56:31.744231 21061 catalog_manager.cc:1347] Generated new cluster ID: 95d1aa5cadac4e7f91e401d311caf87b
I20260927 02:56:31.744283 21061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:31.760027 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:31.743952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43411 (local address 127.0.0.1:35625)
0927 02:56:31.744093 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:31.744103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:31.754964 (+ 10861us) server_negotiation.cc:407] Connection header received
0927 02:56:31.755092 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:31.755099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:31.755155 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:31.755298 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:31.755514 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:31.756858 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:31.757959 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:31.758544 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:31.758629 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:31.759335 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:31.759339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:31.759339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:31.759570 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:31.759607 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:31.759621 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:31.759678 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:31.759744 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:56:31.759809 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260927 02:56:31.760911 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:31.766460 21061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:31.767143 21061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:31.769490 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:31.762269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50639 (local address 127.0.0.1:35625)
0927 02:56:31.762300 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:31.762301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:31.766315 (+ 4014us) server_negotiation.cc:407] Connection header received
0927 02:56:31.766357 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:31.766358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:31.766368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:31.766427 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:31.766638 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:31.767327 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:31.768324 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:31.768605 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:31.768654 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:31.769266 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:31.769268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:31.769268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:31.769271 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:31.769286 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:31.769290 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:31.769334 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:31.769384 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:56:31.769421 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:56:31.772950 21061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff7c624ef4b7481a8509e8b0b9bb6519: Generated new TSK 0
I20260927 02:56:31.773113 21061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:31.780333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 20981
2026-09-27T02:56:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:31 dist-test-slave-mfx9 krb5kdc[21080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:31 dist-test-slave-mfx9 krb5kdc[21080](info): set up 2 sockets
Sep 27 02:56:31 dist-test-slave-mfx9 krb5kdc[21080](info): commencing operation
krb5kdc: starting...
W20260927 02:56:32.714418 392 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:32 dist-test-slave-mfx9 krb5kdc[21080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:32.821702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:32.897632 21093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:32.897825 21093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:32.897856 21093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:32.898974 21093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:32.899019 21093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:32.899037 21093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:32.899053 21093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:32.899068 21093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:32.900502 21093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:32.900717 21093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:32.900908 21093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:32.905387 21099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:32.905375 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:32.905376 21098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:32.905458 21093 server_base.cc:1034] running on GCE node
I20260927 02:56:32.905793 21093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:32.906039 21093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:32.907172 21093 hybrid_clock.cc:648] HybridClock initialized: now 1790477792907162 us; error 28 us; skew 500 ppm
Sep 27 02:56:32 dist-test-slave-mfx9 krb5kdc[21080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:32.909422 21093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:32.909817 21093 webserver.cc:460] Webserver started at http://127.0.0.1:44917/ using document root <none> and password file <none>
I20260927 02:56:32.909993 21093 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:32.910030 21093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:32.910113 21093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:32.911072 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "9f69b3b7623e46c19b4c809174ca1af3"
format_stamp: "Formatted at 2026-09-27 02:56:32 on dist-test-slave-mfx9"
server_key: "f98d8accdfcc53593c74360e802a1ae7"
server_key_iv: "8d247f48eff01d959fd2064351cd39c0"
server_key_version: "encryptionkey@0"
I20260927 02:56:32.911356 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f69b3b7623e46c19b4c809174ca1af3"
format_stamp: "Formatted at 2026-09-27 02:56:32 on dist-test-slave-mfx9"
server_key: "f98d8accdfcc53593c74360e802a1ae7"
server_key_iv: "8d247f48eff01d959fd2064351cd39c0"
server_key_version: "encryptionkey@0"
I20260927 02:56:32.912438 21093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:56:32.913076 21108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:32.913259 21093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:32.913338 21093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "9f69b3b7623e46c19b4c809174ca1af3"
format_stamp: "Formatted at 2026-09-27 02:56:32 on dist-test-slave-mfx9"
server_key: "f98d8accdfcc53593c74360e802a1ae7"
server_key_iv: "8d247f48eff01d959fd2064351cd39c0"
server_key_version: "encryptionkey@0"
I20260927 02:56:32.913393 21093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:32.916610 21093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:32.918277 21093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:32.918380 21093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:32.922075 21093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33899
I20260927 02:56:32.922109 21160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33899 every 8 connection(s)
I20260927 02:56:32.922382 21093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:32.923300 21161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:32.925227 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3: Bootstrap starting.
I20260927 02:56:32.925472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21093
I20260927 02:56:32.925586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:32.925750 392 external_mini_cluster.cc:1428] Setting key d3a7a0e6f5e67973165e1c24aa0030cd
I20260927 02:56:32.925845 21161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:32.926106 21161 log.cc:824] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:32.926730 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3: No bootstrap required, opened a new log
I20260927 02:56:32.928186 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:32.926193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58541 (local address 127.0.0.1:33899)
0927 02:56:32.926509 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:32.926515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:32.926537 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:56:32.926986 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:32.926993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:32.927017 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:32.927136 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:32.927748 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:32.927752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:32.927752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:32.927756 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:32.927808 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:32.927819 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:32.927887 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:32.927942 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 02:56:32.927979 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":88,"threads_started":1}
I20260927 02:56:32.928987 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:32.928903 21161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f69b3b7623e46c19b4c809174ca1af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33899 } }
I20260927 02:56:32.929067 21161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:32.929111 21161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f69b3b7623e46c19b4c809174ca1af3, State: Initialized, Role: FOLLOWER
I20260927 02:56:32.929234 21161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f69b3b7623e46c19b4c809174ca1af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33899 } }
I20260927 02:56:32.929320 21161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:32.929353 21161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:32.929404 21161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:32.929924 21161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f69b3b7623e46c19b4c809174ca1af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33899 } }
I20260927 02:56:32.930029 21161 leader_election.cc:302] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f69b3b7623e46c19b4c809174ca1af3; no voters:
I20260927 02:56:32.930189 21161 leader_election.cc:288] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:32.930235 21170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:32.930392 21170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [term 1 LEADER]: Becoming Leader. State: Replica: 9f69b3b7623e46c19b4c809174ca1af3, State: Running, Role: LEADER
I20260927 02:56:32.930421 21161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:32.930475 21170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f69b3b7623e46c19b4c809174ca1af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33899 } }
I20260927 02:56:32.930751 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f69b3b7623e46c19b4c809174ca1af3. Latest consensus state: current_term: 1 leader_uuid: "9f69b3b7623e46c19b4c809174ca1af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f69b3b7623e46c19b4c809174ca1af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33899 } } }
I20260927 02:56:32.930821 21171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f69b3b7623e46c19b4c809174ca1af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f69b3b7623e46c19b4c809174ca1af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33899 } } }
I20260927 02:56:32.930872 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:32.930922 21171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:32.931097 21178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:32.931758 21178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:32.932979 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:32.930232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48365 (local address 127.0.0.1:33899)
0927 02:56:32.930260 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:32.930261 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:32.931993 (+ 1732us) server_negotiation.cc:407] Connection header received
0927 02:56:32.932102 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:32.932103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:32.932109 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:32.932164 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:32.932769 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:32.932770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:32.932770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:32.932775 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:32.932787 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:32.932791 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:32.932848 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:32.932906 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 02:56:32.932919 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:56:32.933439 21178 catalog_manager.cc:1347] Generated new cluster ID: f7a2c25425004739a5b68839e95145a0
I20260927 02:56:32.933492 21178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:32.960676 21178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:32.960857 21178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:32.963796 21178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f69b3b7623e46c19b4c809174ca1af3: Generated new TSK 0
I20260927 02:56:32.963954 21178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:32.966005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21093
2026-09-27T02:56:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:33 dist-test-slave-mfx9 krb5kdc[21192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:33 dist-test-slave-mfx9 krb5kdc[21192](info): set up 2 sockets
Sep 27 02:56:33 dist-test-slave-mfx9 krb5kdc[21192](info): commencing operation
krb5kdc: starting...
W20260927 02:56:33.906507 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:33 dist-test-slave-mfx9 krb5kdc[21192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:34.011413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:34.088735 21205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:34.088929 21205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:34.088961 21205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:34.090088 21205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:34.090126 21205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:34.090144 21205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:34.090160 21205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:34.090174 21205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:34.091621 21205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:34.091856 21205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:34.092047 21205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:34.096428 21211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:34.096433 21210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:34.096489 21213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:34.096617 21205 server_base.cc:1034] running on GCE node
I20260927 02:56:34.096962 21205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:34.097225 21205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:34.098382 21205 hybrid_clock.cc:648] HybridClock initialized: now 1790477794098368 us; error 28 us; skew 500 ppm
Sep 27 02:56:34 dist-test-slave-mfx9 krb5kdc[21192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:34.104281 21205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:34.104665 21205 webserver.cc:460] Webserver started at http://127.0.0.1:35751/ using document root <none> and password file <none>
I20260927 02:56:34.104835 21205 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:34.104890 21205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:34.104998 21205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:34.105944 21205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "c804997a3a7241138d0dad63eba546d9"
format_stamp: "Formatted at 2026-09-27 02:56:34 on dist-test-slave-mfx9"
server_key: "7f02c9f07b13c4d308d8b9775ebac86e"
server_key_iv: "14cf12e39d56751550cca7d3bb697f5f"
server_key_version: "encryptionkey@0"
I20260927 02:56:34.106238 21205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "c804997a3a7241138d0dad63eba546d9"
format_stamp: "Formatted at 2026-09-27 02:56:34 on dist-test-slave-mfx9"
server_key: "7f02c9f07b13c4d308d8b9775ebac86e"
server_key_iv: "14cf12e39d56751550cca7d3bb697f5f"
server_key_version: "encryptionkey@0"
I20260927 02:56:34.107398 21205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:56:34.108001 21220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:34.108137 21205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:34.108208 21205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "c804997a3a7241138d0dad63eba546d9"
format_stamp: "Formatted at 2026-09-27 02:56:34 on dist-test-slave-mfx9"
server_key: "7f02c9f07b13c4d308d8b9775ebac86e"
server_key_iv: "14cf12e39d56751550cca7d3bb697f5f"
server_key_version: "encryptionkey@0"
I20260927 02:56:34.108266 21205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:34.120251 21205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:34.121943 21205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:34.122052 21205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:34.125566 21205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41939
I20260927 02:56:34.125593 21272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41939 every 8 connection(s)
I20260927 02:56:34.125880 21205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:34.126603 21273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:34.128665 21273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9: Bootstrap starting.
I20260927 02:56:34.129175 21273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:34.129411 21273 log.cc:824] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:34.129997 21273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9: No bootstrap required, opened a new log
I20260927 02:56:34.131539 21273 raft_consensus.cc:348] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c804997a3a7241138d0dad63eba546d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260927 02:56:34.131668 21273 raft_consensus.cc:374] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:34.131690 21273 raft_consensus.cc:729] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c804997a3a7241138d0dad63eba546d9, State: Initialized, Role: FOLLOWER
I20260927 02:56:34.131822 21273 consensus_queue.cc:260] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c804997a3a7241138d0dad63eba546d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260927 02:56:34.131879 21273 raft_consensus.cc:388] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:34.131910 21273 raft_consensus.cc:482] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:34.131955 21273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:34.132414 21273 raft_consensus.cc:504] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c804997a3a7241138d0dad63eba546d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260927 02:56:34.132496 21273 leader_election.cc:302] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c804997a3a7241138d0dad63eba546d9; no voters:
I20260927 02:56:34.132642 21273 leader_election.cc:288] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:34.132675 21276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:34.132795 21276 raft_consensus.cc:686] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [term 1 LEADER]: Becoming Leader. State: Replica: c804997a3a7241138d0dad63eba546d9, State: Running, Role: LEADER
I20260927 02:56:34.132874 21273 sys_catalog.cc:564] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:34.132884 21276 consensus_queue.cc:237] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c804997a3a7241138d0dad63eba546d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260927 02:56:34.133239 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c804997a3a7241138d0dad63eba546d9. Latest consensus state: current_term: 1 leader_uuid: "c804997a3a7241138d0dad63eba546d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c804997a3a7241138d0dad63eba546d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } } }
I20260927 02:56:34.133333 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:34.133359 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c804997a3a7241138d0dad63eba546d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c804997a3a7241138d0dad63eba546d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } } }
I20260927 02:56:34.133417 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:34.133526 21284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:34.133982 21284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:34.135365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21205
I20260927 02:56:34.135474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:34.135464 21284 catalog_manager.cc:1347] Generated new cluster ID: 81e82eb453e3469f9232bf5c0fb60fe7
I20260927 02:56:34.135509 21284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:34.135641 392 external_mini_cluster.cc:1428] Setting key 5528e3da5139eef922f2935d7490e244
Sep 27 02:56:34 dist-test-slave-mfx9 krb5kdc[21192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:56:34.146205 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:34.136052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:53421 (local address 127.0.0.1:41939)
0927 02:56:34.136184 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:34.136192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:34.136255 (+ 63us) server_negotiation.cc:407] Connection header received
0927 02:56:34.136360 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:34.136366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:34.136417 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:34.136527 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:34.137221 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:34.138594 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:34.139662 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:34.140246 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:34.140333 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:34.142951 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:34.142954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:34.142955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:34.142967 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:34.144717 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:34.145098 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:34.145100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:34.145101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:34.145200 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:34.145389 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:34.145390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:34.145390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:34.145668 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:34.145807 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:34.145903 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:56:34.145996 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260927 02:56:34.147050 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:34.154834 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:34.148365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:52049 (local address 127.0.0.1:41939)
0927 02:56:34.148408 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:34.148410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:34.148521 (+ 111us) server_negotiation.cc:407] Connection header received
0927 02:56:34.148576 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:34.148577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:34.148586 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:34.148630 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:34.149153 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:34.149846 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:34.150822 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:34.151112 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:34.151163 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:34.152379 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:34.152380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:34.152380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:34.152386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:34.153751 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:34.154115 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:34.154116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:34.154116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:34.154160 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:34.154365 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:34.154366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:34.154366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:34.154480 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:34.154554 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:34.154678 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 02:56:34.154732 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:56:34.162900 21284 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:34.163390 21284 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:34.169822 21284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c804997a3a7241138d0dad63eba546d9: Generated new TSK 0
I20260927 02:56:34.169986 21284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:34.180182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21205
W20260927 02:56:34.183998 21190 connection.cc:535] client connection to 127.0.0.1:41939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41939): Connection reset by peer (error 104)
2026-09-27T02:56:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:34 dist-test-slave-mfx9 krb5kdc[21304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:34 dist-test-slave-mfx9 krb5kdc[21304](info): set up 2 sockets
Sep 27 02:56:34 dist-test-slave-mfx9 krb5kdc[21304](info): commencing operation
krb5kdc: starting...
W20260927 02:56:35.126508 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:35 dist-test-slave-mfx9 krb5kdc[21304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:35.234529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:35.310935 21317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:35.311115 21317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:35.311144 21317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:35.312243 21317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:35.312281 21317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:35.312299 21317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:35.312315 21317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:35.312330 21317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:35.313714 21317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:35.313941 21317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:35.314121 21317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:35.318483 21323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:35.318493 21322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:35.318493 21325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:35.318672 21317 server_base.cc:1034] running on GCE node
I20260927 02:56:35.318914 21317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:35.319162 21317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:35.320297 21317 hybrid_clock.cc:648] HybridClock initialized: now 1790477795320285 us; error 26 us; skew 500 ppm
Sep 27 02:56:35 dist-test-slave-mfx9 krb5kdc[21304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:35.322507 21317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:35.322928 21317 webserver.cc:460] Webserver started at http://127.0.0.1:39069/ using document root <none> and password file <none>
I20260927 02:56:35.323105 21317 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:35.323141 21317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:35.323225 21317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:35.324129 21317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "d22ba8d719ae4f59bce28279560b8b37"
format_stamp: "Formatted at 2026-09-27 02:56:35 on dist-test-slave-mfx9"
server_key: "1b96deb84e448cc8b7d6c6b9779e75b7"
server_key_iv: "8e95a71a9cf013b18b692a6b01d9303c"
server_key_version: "encryptionkey@0"
I20260927 02:56:35.324398 21317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "d22ba8d719ae4f59bce28279560b8b37"
format_stamp: "Formatted at 2026-09-27 02:56:35 on dist-test-slave-mfx9"
server_key: "1b96deb84e448cc8b7d6c6b9779e75b7"
server_key_iv: "8e95a71a9cf013b18b692a6b01d9303c"
server_key_version: "encryptionkey@0"
I20260927 02:56:35.325486 21317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:56:35.326092 21332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:35.326227 21317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:35.326308 21317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "d22ba8d719ae4f59bce28279560b8b37"
format_stamp: "Formatted at 2026-09-27 02:56:35 on dist-test-slave-mfx9"
server_key: "1b96deb84e448cc8b7d6c6b9779e75b7"
server_key_iv: "8e95a71a9cf013b18b692a6b01d9303c"
server_key_version: "encryptionkey@0"
I20260927 02:56:35.326360 21317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:35.348871 21317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:35.350495 21317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:35.350596 21317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:35.354367 21317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36333
I20260927 02:56:35.354408 21384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36333 every 8 connection(s)
I20260927 02:56:35.354692 21317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:35.355520 21385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:35.357609 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37: Bootstrap starting.
I20260927 02:56:35.358124 21385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:35.358326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21317
I20260927 02:56:35.358373 21385 log.cc:824] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:35.358428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:35.358588 392 external_mini_cluster.cc:1428] Setting key 31bcf492646ea6e29dfcec935db45f9d
I20260927 02:56:35.359081 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37: No bootstrap required, opened a new log
I20260927 02:56:35.361205 21385 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22ba8d719ae4f59bce28279560b8b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36333 } }
I20260927 02:56:35.361336 21385 raft_consensus.cc:374] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:35.361358 21385 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22ba8d719ae4f59bce28279560b8b37, State: Initialized, Role: FOLLOWER
I20260927 02:56:35.361461 21385 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22ba8d719ae4f59bce28279560b8b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36333 } }
I20260927 02:56:35.361532 21385 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:35.361560 21385 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:35.361606 21385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:35.362239 21385 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22ba8d719ae4f59bce28279560b8b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36333 } }
I20260927 02:56:35.362345 21385 leader_election.cc:302] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22ba8d719ae4f59bce28279560b8b37; no voters:
I20260927 02:56:35.362506 21385 leader_election.cc:288] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:35.362543 21390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:35.362707 21390 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [term 1 LEADER]: Becoming Leader. State: Replica: d22ba8d719ae4f59bce28279560b8b37, State: Running, Role: LEADER
I20260927 02:56:35.362826 21385 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:35.362831 21390 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22ba8d719ae4f59bce28279560b8b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36333 } }
I20260927 02:56:35.363274 21391 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d22ba8d719ae4f59bce28279560b8b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22ba8d719ae4f59bce28279560b8b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36333 } } }
I20260927 02:56:35.363461 21391 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:35.363401 21392 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22ba8d719ae4f59bce28279560b8b37. Latest consensus state: current_term: 1 leader_uuid: "d22ba8d719ae4f59bce28279560b8b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22ba8d719ae4f59bce28279560b8b37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36333 } } }
I20260927 02:56:35.363693 21392 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:35.363698 21399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:35.364217 21399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:35.365712 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:35.359042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:36059 (local address 127.0.0.1:36333)
0927 02:56:35.359247 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:35.359254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:35.359282 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:56:35.360162 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:35.360169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:35.360223 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:35.360344 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:35.360575 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:35.361492 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:35.362532 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:35.363110 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:35.363207 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:35.365082 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:35.365086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:35.365087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:35.365286 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:35.365321 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:35.365335 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:35.365412 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:35.365482 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:56:35.365506 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260927 02:56:35.366544 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:35.370995 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:35.367848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:40473 (local address 127.0.0.1:36333)
0927 02:56:35.367884 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:35.367886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:35.367930 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:56:35.367996 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:35.367996 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:35.368003 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:35.368043 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:35.368155 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:35.368592 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:35.369592 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:35.369945 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:35.370005 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:35.370687 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:35.370688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:35.370688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:35.370692 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:35.370704 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:35.370708 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:35.370782 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:35.370877 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 02:56:35.370886 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:56:35.375156 21399 catalog_manager.cc:1347] Generated new cluster ID: b71d8b0cffdc4aacb0617fc838a17a1e
I20260927 02:56:35.375207 21399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:35.395052 21399 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:35.395550 21399 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:35.399072 21399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d22ba8d719ae4f59bce28279560b8b37: Generated new TSK 0
I20260927 02:56:35.399214 21399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:35.404902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21317
W20260927 02:56:35.408417 21302 connection.cc:535] client connection to 127.0.0.1:36333 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36333): Connection reset by peer (error 104)
2026-09-27T02:56:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:35 dist-test-slave-mfx9 krb5kdc[21416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:35 dist-test-slave-mfx9 krb5kdc[21416](info): set up 2 sockets
Sep 27 02:56:35 dist-test-slave-mfx9 krb5kdc[21416](info): commencing operation
krb5kdc: starting...
W20260927 02:56:36.350479 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:36 dist-test-slave-mfx9 krb5kdc[21416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:36.457847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:36.534413 21429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:36.534602 21429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:36.534623 21429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:36.535775 21429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:36.535815 21429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:36.535828 21429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:36.535841 21429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:36.535864 21429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:36.537293 21429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:36.537508 21429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:36.537715 21429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:36.542229 21434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:36.542330 21435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:36.542356 21437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:36.542490 21429 server_base.cc:1034] running on GCE node
I20260927 02:56:36.542704 21429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:36.542986 21429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:36.544121 21429 hybrid_clock.cc:648] HybridClock initialized: now 1790477796544112 us; error 33 us; skew 500 ppm
Sep 27 02:56:36 dist-test-slave-mfx9 krb5kdc[21416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:36.546329 21429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:36.546708 21429 webserver.cc:460] Webserver started at http://127.0.0.1:34625/ using document root <none> and password file <none>
I20260927 02:56:36.546921 21429 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:36.546967 21429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:36.547086 21429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:36.548069 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9"
format_stamp: "Formatted at 2026-09-27 02:56:36 on dist-test-slave-mfx9"
server_key: "19a58451dc5f8d497e0e9eaf6a31afde"
server_key_iv: "b20613b8c0a9ea281bff2a29edfe7cb3"
server_key_version: "encryptionkey@0"
I20260927 02:56:36.548384 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9"
format_stamp: "Formatted at 2026-09-27 02:56:36 on dist-test-slave-mfx9"
server_key: "19a58451dc5f8d497e0e9eaf6a31afde"
server_key_iv: "b20613b8c0a9ea281bff2a29edfe7cb3"
server_key_version: "encryptionkey@0"
I20260927 02:56:36.549558 21429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:36.550148 21444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:36.550283 21429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:36.550359 21429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9"
format_stamp: "Formatted at 2026-09-27 02:56:36 on dist-test-slave-mfx9"
server_key: "19a58451dc5f8d497e0e9eaf6a31afde"
server_key_iv: "b20613b8c0a9ea281bff2a29edfe7cb3"
server_key_version: "encryptionkey@0"
I20260927 02:56:36.550415 21429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:36.553444 21429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:36.555043 21429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:36.555157 21429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:36.558638 21429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35227
I20260927 02:56:36.558665 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35227 every 8 connection(s)
I20260927 02:56:36.558966 21429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:36.559813 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:36.561474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21429
I20260927 02:56:36.561580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:36.561753 392 external_mini_cluster.cc:1428] Setting key 338fae7bf675a7635424b485401b85f4
I20260927 02:56:36.562053 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9: Bootstrap starting.
I20260927 02:56:36.562624 21497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:36.562892 21497 log.cc:824] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:36.563575 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9: No bootstrap required, opened a new log
I20260927 02:56:36.564049 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:36.562224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:41253 (local address 127.0.0.1:35227)
0927 02:56:36.562381 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:36.562389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:36.562413 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:56:36.562906 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:36.562914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:36.562942 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:36.563079 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:36.563641 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:36.563645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:36.563645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:36.563649 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:36.563696 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:36.563711 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:36.563780 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:36.563829 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:56:36.563868 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260927 02:56:36.564898 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:36.565671 21497 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35227 } }
I20260927 02:56:36.565840 21497 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:36.565886 21497 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fe6f0a0335420c9a1fa8d3761d1bc9, State: Initialized, Role: FOLLOWER
I20260927 02:56:36.566011 21497 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35227 } }
I20260927 02:56:36.566102 21497 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:36.566139 21497 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:36.566183 21497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:36.566807 21497 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35227 } }
I20260927 02:56:36.566905 21497 leader_election.cc:302] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2fe6f0a0335420c9a1fa8d3761d1bc9; no voters:
I20260927 02:56:36.567050 21497 leader_election.cc:288] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:36.567116 21507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:36.567284 21497 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:36.567293 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:36.566330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:37295 (local address 127.0.0.1:35227)
0927 02:56:36.566362 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:36.566363 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:36.566470 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:56:36.566537 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:36.566537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:36.566541 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:36.566580 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:36.567102 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:36.567103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:36.567103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:36.567107 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:36.567120 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:36.567124 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:36.567176 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:36.567230 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 02:56:36.567243 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:56:36.567625 21507 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [term 1 LEADER]: Becoming Leader. State: Replica: a2fe6f0a0335420c9a1fa8d3761d1bc9, State: Running, Role: LEADER
I20260927 02:56:36.567806 21507 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35227 } }
I20260927 02:56:36.568173 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35227 } } }
I20260927 02:56:36.568190 21509 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2fe6f0a0335420c9a1fa8d3761d1bc9. Latest consensus state: current_term: 1 leader_uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fe6f0a0335420c9a1fa8d3761d1bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35227 } } }
I20260927 02:56:36.568291 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:36.568298 21509 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:36.568490 21521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:36.569005 21521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:36.570297 21521 catalog_manager.cc:1347] Generated new cluster ID: e29354f522684ba7a0e59b91aaec2f8e
I20260927 02:56:36.570343 21521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:36.580329 21521 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:36.580474 21521 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:36.588335 21521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2fe6f0a0335420c9a1fa8d3761d1bc9: Generated new TSK 0
I20260927 02:56:36.588480 21521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:36.595974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21429
2026-09-27T02:56:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1191 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:36 dist-test-slave-mfx9 krb5kdc[21528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:36 dist-test-slave-mfx9 krb5kdc[21528](info): set up 2 sockets
Sep 27 02:56:36 dist-test-slave-mfx9 krb5kdc[21528](info): commencing operation
krb5kdc: starting...
W20260927 02:56:37.538552 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:37 dist-test-slave-mfx9 krb5kdc[21528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:37.643002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:37.718550 21541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:37.718750 21541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:37.718796 21541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:37.719910 21541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:37.719949 21541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:37.719972 21541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:37.719987 21541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:37.720003 21541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:37.721422 21541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:37.721647 21541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:37.721848 21541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:37.726212 21546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:37.726226 21547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:37.726255 21541 server_base.cc:1034] running on GCE node
W20260927 02:56:37.726226 21549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:37.726624 21541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:37.726905 21541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:37.728036 21541 hybrid_clock.cc:648] HybridClock initialized: now 1790477797728026 us; error 29 us; skew 500 ppm
Sep 27 02:56:37 dist-test-slave-mfx9 krb5kdc[21528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:37.730183 21541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:37.730531 21541 webserver.cc:460] Webserver started at http://127.0.0.1:34835/ using document root <none> and password file <none>
I20260927 02:56:37.730701 21541 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:37.730746 21541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:37.730878 21541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:37.731781 21541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb"
format_stamp: "Formatted at 2026-09-27 02:56:37 on dist-test-slave-mfx9"
server_key: "67a076201726eab4ad31449d80c0399f"
server_key_iv: "4b75b6d324bdac4de4853c6f9d907594"
server_key_version: "encryptionkey@0"
I20260927 02:56:37.732052 21541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb"
format_stamp: "Formatted at 2026-09-27 02:56:37 on dist-test-slave-mfx9"
server_key: "67a076201726eab4ad31449d80c0399f"
server_key_iv: "4b75b6d324bdac4de4853c6f9d907594"
server_key_version: "encryptionkey@0"
I20260927 02:56:37.733142 21541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:37.733729 21556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:37.733906 21541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:37.733982 21541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb"
format_stamp: "Formatted at 2026-09-27 02:56:37 on dist-test-slave-mfx9"
server_key: "67a076201726eab4ad31449d80c0399f"
server_key_iv: "4b75b6d324bdac4de4853c6f9d907594"
server_key_version: "encryptionkey@0"
I20260927 02:56:37.734040 21541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:37.756040 21541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:37.757673 21541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:37.757807 21541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:37.761358 21541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44793
I20260927 02:56:37.761390 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44793 every 8 connection(s)
I20260927 02:56:37.761689 21541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:37.762457 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:37.764649 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb: Bootstrap starting.
I20260927 02:56:37.765142 21609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:37.765352 21609 log.cc:824] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:37.765961 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb: No bootstrap required, opened a new log
I20260927 02:56:37.766717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21541
I20260927 02:56:37.766860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:37.767048 392 external_mini_cluster.cc:1428] Setting key 4d8a5c0a3d0cc09e871b6eb7aaea13b5
I20260927 02:56:37.767861 21609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } }
I20260927 02:56:37.768009 21609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:37.768044 21609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ced759a8d74fe9ab9e1ed3c6ba1ffb, State: Initialized, Role: FOLLOWER
I20260927 02:56:37.768167 21609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } }
I20260927 02:56:37.768244 21609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:37.768280 21609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:37.768337 21609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:37.769222 21609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } }
I20260927 02:56:37.769317 21609 leader_election.cc:302] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ced759a8d74fe9ab9e1ed3c6ba1ffb; no voters:
I20260927 02:56:37.769449 21609 leader_election.cc:288] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:37.769496 21614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:37.769666 21614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [term 1 LEADER]: Becoming Leader. State: Replica: 89ced759a8d74fe9ab9e1ed3c6ba1ffb, State: Running, Role: LEADER
I20260927 02:56:37.769673 21609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:37.769800 21614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } }
I20260927 02:56:37.770285 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ced759a8d74fe9ab9e1ed3c6ba1ffb. Latest consensus state: current_term: 1 leader_uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } } }
I20260927 02:56:37.770368 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:37.770489 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ced759a8d74fe9ab9e1ed3c6ba1ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } } }
I20260927 02:56:37.770525 21622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:37.770546 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:37.771068 21622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:37.772598 21622 catalog_manager.cc:1347] Generated new cluster ID: 2c5d87b71a654639a1412f4565f8ecd8
I20260927 02:56:37.772642 21622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:56:37 dist-test-slave-mfx9 krb5kdc[21528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:56:37.777805 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:37.767514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:52967 (local address 127.0.0.1:44793)
0927 02:56:37.767658 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:37.767665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:37.767691 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:56:37.767822 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:37.767827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:37.767877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:37.767975 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:37.768709 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:37.769743 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:37.770855 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:37.771571 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:37.771675 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:37.774311 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:37.774315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:37.774315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:37.774327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:37.776160 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:37.776576 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:37.776579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:37.776579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:37.776691 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:37.776908 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:37.776909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:37.776909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:37.777168 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:37.777306 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:37.777431 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 02:56:37.777612 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260927 02:56:37.778641 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:37.786478 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:37.779958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:48679 (local address 127.0.0.1:44793)
0927 02:56:37.779998 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:37.780000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:37.780089 (+ 89us) server_negotiation.cc:407] Connection header received
0927 02:56:37.780134 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:37.780135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:37.780145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:37.780204 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:37.780744 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:37.781503 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:37.782411 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:37.782743 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:37.782827 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:37.783997 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:37.783998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:37.783999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:37.784005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:37.785438 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:37.785799 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:37.785800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:37.785800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:37.785853 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:37.786030 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:37.786031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:37.786031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:37.786147 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:37.786259 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:37.786336 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:56:37.786393 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:56:37.795795 21622 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:37.796259 21622 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:37.799780 21622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89ced759a8d74fe9ab9e1ed3c6ba1ffb: Generated new TSK 0
I20260927 02:56:37.799921 21622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:37.802634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21541
W20260927 02:56:37.806427 21526 connection.cc:535] client connection to 127.0.0.1:44793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44793): Connection reset by peer (error 104)
2026-09-27T02:56:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:37 dist-test-slave-mfx9 krb5kdc[21640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:37 dist-test-slave-mfx9 krb5kdc[21640](info): set up 2 sockets
Sep 27 02:56:37 dist-test-slave-mfx9 krb5kdc[21640](info): commencing operation
krb5kdc: starting...
W20260927 02:56:38.746356 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:38 dist-test-slave-mfx9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:38.854220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:38.931031 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:38.931221 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:38.931241 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:38.932303 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:38.932336 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:38.932349 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:38.932360 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:38.932370 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:38.933745 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:38.933924 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:38.934091 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:38.938707 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:38.938684 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:38.938750 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:38.938827 21653 server_base.cc:1034] running on GCE node
I20260927 02:56:38.939157 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:38.939392 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:38.940532 21653 hybrid_clock.cc:648] HybridClock initialized: now 1790477798940509 us; error 38 us; skew 500 ppm
Sep 27 02:56:38 dist-test-slave-mfx9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:38.949258 21653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:38.949640 21653 webserver.cc:460] Webserver started at http://127.0.0.1:45873/ using document root <none> and password file <none>
I20260927 02:56:38.949810 21653 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:38.949846 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:38.949930 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:38.950918 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "a24d6f31a8734d77b859a24ac9259806"
format_stamp: "Formatted at 2026-09-27 02:56:38 on dist-test-slave-mfx9"
server_key: "079cada8940b1e0396a3a3845a7914f8"
server_key_iv: "2bc5d7db3c3b58946fab5133eb50ddb2"
server_key_version: "encryptionkey@0"
I20260927 02:56:38.951207 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "a24d6f31a8734d77b859a24ac9259806"
format_stamp: "Formatted at 2026-09-27 02:56:38 on dist-test-slave-mfx9"
server_key: "079cada8940b1e0396a3a3845a7914f8"
server_key_iv: "2bc5d7db3c3b58946fab5133eb50ddb2"
server_key_version: "encryptionkey@0"
I20260927 02:56:38.952366 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:56:38.953074 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:38.953239 21653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:38.953318 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "a24d6f31a8734d77b859a24ac9259806"
format_stamp: "Formatted at 2026-09-27 02:56:38 on dist-test-slave-mfx9"
server_key: "079cada8940b1e0396a3a3845a7914f8"
server_key_iv: "2bc5d7db3c3b58946fab5133eb50ddb2"
server_key_version: "encryptionkey@0"
I20260927 02:56:38.953379 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:38.960155 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:38.961709 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:38.961807 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:38.965312 21653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41325
I20260927 02:56:38.965354 21720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41325 every 8 connection(s)
I20260927 02:56:38.965627 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:38.966547 21721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:38.968161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21653
I20260927 02:56:38.968273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:38.968461 392 external_mini_cluster.cc:1428] Setting key 2db68782be213429bc8989ae70533ed2
I20260927 02:56:38.969023 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806: Bootstrap starting.
I20260927 02:56:38.969702 21721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:38.970022 21721 log.cc:824] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:38.970674 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806: No bootstrap required, opened a new log
I20260927 02:56:38.972396 21721 raft_consensus.cc:348] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24d6f31a8734d77b859a24ac9259806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260927 02:56:38.972532 21721 raft_consensus.cc:374] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:38.972568 21721 raft_consensus.cc:729] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24d6f31a8734d77b859a24ac9259806, State: Initialized, Role: FOLLOWER
I20260927 02:56:38.972710 21721 consensus_queue.cc:260] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24d6f31a8734d77b859a24ac9259806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260927 02:56:38.972798 21721 raft_consensus.cc:388] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:38.972838 21721 raft_consensus.cc:482] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:38.972890 21721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:38.973410 21721 raft_consensus.cc:504] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24d6f31a8734d77b859a24ac9259806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260927 02:56:38.973507 21721 leader_election.cc:302] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a24d6f31a8734d77b859a24ac9259806; no voters:
I20260927 02:56:38.973671 21721 leader_election.cc:288] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:38.973693 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:38.968949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:35901 (local address 127.0.0.1:41325)
0927 02:56:38.969113 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:38.969131 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:38.969275 (+ 144us) server_negotiation.cc:407] Connection header received
0927 02:56:38.969837 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:38.969847 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:38.969918 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:38.970080 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:38.970277 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:38.971191 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:38.971965 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:38.972355 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:38.972453 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:38.973114 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:38.973118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:38.973118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:38.973286 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:38.973314 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:38.973324 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:38.973391 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:38.973480 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:56:38.973508 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260927 02:56:38.973714 21726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:38.973855 21726 raft_consensus.cc:686] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [term 1 LEADER]: Becoming Leader. State: Replica: a24d6f31a8734d77b859a24ac9259806, State: Running, Role: LEADER
I20260927 02:56:38.973924 21721 sys_catalog.cc:564] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:38.973933 21726 consensus_queue.cc:237] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24d6f31a8734d77b859a24ac9259806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260927 02:56:38.974264 21728 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a24d6f31a8734d77b859a24ac9259806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24d6f31a8734d77b859a24ac9259806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } } }
I20260927 02:56:38.974272 21726 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a24d6f31a8734d77b859a24ac9259806. Latest consensus state: current_term: 1 leader_uuid: "a24d6f31a8734d77b859a24ac9259806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24d6f31a8734d77b859a24ac9259806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } } }
I20260927 02:56:38.974498 21726 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:38.974740 21735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:38.974486 21728 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:38.974879 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:38.975283 21735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:38.979315 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:38.976330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:48965 (local address 127.0.0.1:41325)
0927 02:56:38.976363 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:38.976365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:38.976555 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:56:38.976617 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:38.976618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:38.976626 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:38.976669 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:38.976799 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:38.977242 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:38.978163 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:38.978369 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:38.978416 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:38.979073 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:38.979074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:38.979074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:38.979079 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:38.979092 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:38.979096 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:38.979154 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:38.979242 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 02:56:38.979252 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:38.979969 21735 catalog_manager.cc:1347] Generated new cluster ID: de31430cb2384667a8112a209d99d644
I20260927 02:56:38.980016 21735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:38.993985 21735 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:38.994663 21735 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:39.001008 21735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a24d6f31a8734d77b859a24ac9259806: Generated new TSK 0
I20260927 02:56:39.001165 21735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:39.004314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21653
W20260927 02:56:39.008041 21638 connection.cc:535] client connection to 127.0.0.1:41325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41325): Connection reset by peer (error 104)
2026-09-27T02:56:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:39 dist-test-slave-mfx9 krb5kdc[21752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:39 dist-test-slave-mfx9 krb5kdc[21752](info): set up 2 sockets
Sep 27 02:56:39 dist-test-slave-mfx9 krb5kdc[21752](info): commencing operation
krb5kdc: starting...
W20260927 02:56:39.950562 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:40 dist-test-slave-mfx9 krb5kdc[21752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:56:40.055536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:40.132473 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:40.132658 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:40.132680 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:40.133798 21765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:40.133836 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:40.133849 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:40.133862 21765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:40.133885 21765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:40.135339 21765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:40.135547 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:40.135741 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:40.139997 21770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:40.140015 21773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:40.140095 21765 server_base.cc:1034] running on GCE node
W20260927 02:56:40.140113 21771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:40.140437 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:40.140736 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:40.141891 21765 hybrid_clock.cc:648] HybridClock initialized: now 1790477800141875 us; error 29 us; skew 500 ppm
Sep 27 02:56:40 dist-test-slave-mfx9 krb5kdc[21752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:40.151500 21765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:56:40.151921 21765 webserver.cc:460] Webserver started at http://127.0.0.1:44841/ using document root <none> and password file <none>
I20260927 02:56:40.152107 21765 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:40.152151 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:40.152269 21765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:40.153260 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "db758c53c0f44a709a66f218a8b37f73"
format_stamp: "Formatted at 2026-09-27 02:56:40 on dist-test-slave-mfx9"
server_key: "4ec13bb0a2a0c5414bc0bd27420af0c3"
server_key_iv: "63fdc8572511ac858841791693be27f9"
server_key_version: "encryptionkey@0"
I20260927 02:56:40.153574 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "db758c53c0f44a709a66f218a8b37f73"
format_stamp: "Formatted at 2026-09-27 02:56:40 on dist-test-slave-mfx9"
server_key: "4ec13bb0a2a0c5414bc0bd27420af0c3"
server_key_iv: "63fdc8572511ac858841791693be27f9"
server_key_version: "encryptionkey@0"
I20260927 02:56:40.154850 21765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:40.155524 21780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:40.155663 21765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:40.155777 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "db758c53c0f44a709a66f218a8b37f73"
format_stamp: "Formatted at 2026-09-27 02:56:40 on dist-test-slave-mfx9"
server_key: "4ec13bb0a2a0c5414bc0bd27420af0c3"
server_key_iv: "63fdc8572511ac858841791693be27f9"
server_key_version: "encryptionkey@0"
I20260927 02:56:40.155835 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:40.158881 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:40.160920 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:40.161032 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:40.164552 21765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43423
I20260927 02:56:40.164593 21832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43423 every 8 connection(s)
I20260927 02:56:40.164853 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:40.165705 21833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:40.167752 21833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73: Bootstrap starting.
I20260927 02:56:40.168220 21833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:40.168440 21833 log.cc:824] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:40.168991 21833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73: No bootstrap required, opened a new log
I20260927 02:56:40.169214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21765
I20260927 02:56:40.169305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:40.169477 392 external_mini_cluster.cc:1428] Setting key 64eb119a888aef6b61ea970d6820dae9
I20260927 02:56:40.170859 21833 raft_consensus.cc:348] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db758c53c0f44a709a66f218a8b37f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43423 } }
I20260927 02:56:40.170995 21833 raft_consensus.cc:374] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:40.171031 21833 raft_consensus.cc:729] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db758c53c0f44a709a66f218a8b37f73, State: Initialized, Role: FOLLOWER
I20260927 02:56:40.171195 21833 consensus_queue.cc:260] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db758c53c0f44a709a66f218a8b37f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43423 } }
I20260927 02:56:40.171291 21833 raft_consensus.cc:388] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:40.171321 21833 raft_consensus.cc:482] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:40.171382 21833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:40.171933 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:40.169938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:45405 (local address 127.0.0.1:43423)
0927 02:56:40.170083 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:40.170092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:40.170115 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:56:40.170829 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:40.170835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:40.170860 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:40.170968 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:40.171553 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:40.171556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:40.171557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:40.171561 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:40.171600 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:40.171612 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:40.171666 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:40.171734 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:56:40.171773 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260927 02:56:40.172284 21833 raft_consensus.cc:504] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db758c53c0f44a709a66f218a8b37f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43423 } }
I20260927 02:56:40.172380 21833 leader_election.cc:302] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db758c53c0f44a709a66f218a8b37f73; no voters:
I20260927 02:56:40.172519 21833 leader_election.cc:288] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:40.172549 21838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:40.172672 21838 raft_consensus.cc:686] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [term 1 LEADER]: Becoming Leader. State: Replica: db758c53c0f44a709a66f218a8b37f73, State: Running, Role: LEADER
I20260927 02:56:40.172736 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:40.172742 21833 sys_catalog.cc:564] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:40.172747 21838 consensus_queue.cc:237] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db758c53c0f44a709a66f218a8b37f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43423 } }
I20260927 02:56:40.173053 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db758c53c0f44a709a66f218a8b37f73. Latest consensus state: current_term: 1 leader_uuid: "db758c53c0f44a709a66f218a8b37f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db758c53c0f44a709a66f218a8b37f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43423 } } }
I20260927 02:56:40.173110 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:40.173254 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db758c53c0f44a709a66f218a8b37f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db758c53c0f44a709a66f218a8b37f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43423 } } }
I20260927 02:56:40.173303 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73 [sys.catalog]: This master's current role is: LEADER
W20260927 02:56:40.174434 21857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:40.174517 21857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:40.174576 21844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:40.174722 21844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:40.175468 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:40.174368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:43955 (local address 127.0.0.1:43423)
0927 02:56:40.174395 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:40.174397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:40.174543 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:56:40.174642 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:40.174642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:40.174647 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:40.174684 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:40.175272 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:40.175273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:40.175273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:40.175277 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:40.175289 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:40.175293 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:40.175321 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:40.175402 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:56:40.175414 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11}
I20260927 02:56:40.176122 21844 catalog_manager.cc:1347] Generated new cluster ID: 6520b813068640cd90ec742249a7b600
I20260927 02:56:40.176174 21844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:40.197868 21844 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:40.198021 21844 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:40.203284 21844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db758c53c0f44a709a66f218a8b37f73: Generated new TSK 0
I20260927 02:56:40.203454 21844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:40.209708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21765
2026-09-27T02:56:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:40 dist-test-slave-mfx9 krb5kdc[21864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:40 dist-test-slave-mfx9 krb5kdc[21864](info): set up 2 sockets
Sep 27 02:56:40 dist-test-slave-mfx9 krb5kdc[21864](info): commencing operation
krb5kdc: starting...
W20260927 02:56:41.154466 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:41 dist-test-slave-mfx9 krb5kdc[21864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:41.263270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46075
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:41.339661 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:41.339859 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:41.339890 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:41.341010 21877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:41.341049 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:41.341068 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:41.341084 21877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:41.341099 21877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:41.342495 21877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:41.342717 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:41.342923 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:41.347540 21883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:41.347499 21882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:41.347594 21885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:41.347617 21877 server_base.cc:1034] running on GCE node
I20260927 02:56:41.348003 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:41.348268 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:41.349412 21877 hybrid_clock.cc:648] HybridClock initialized: now 1790477801349402 us; error 29 us; skew 500 ppm
Sep 27 02:56:41 dist-test-slave-mfx9 krb5kdc[21864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:41.351756 21877 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:41.352151 21877 webserver.cc:460] Webserver started at http://127.0.98.62:37313/ using document root <none> and password file <none>
I20260927 02:56:41.352311 21877 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:41.352347 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:41.352433 21877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:41.353354 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "f6f15d706dec4be2bed71671cec2b974"
format_stamp: "Formatted at 2026-09-27 02:56:41 on dist-test-slave-mfx9"
server_key: "352f8380c0b11576057e26e098c6d3b3"
server_key_iv: "6b100b5e016d15c6516de020c54e0b75"
server_key_version: "encryptionkey@0"
I20260927 02:56:41.353626 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6f15d706dec4be2bed71671cec2b974"
format_stamp: "Formatted at 2026-09-27 02:56:41 on dist-test-slave-mfx9"
server_key: "352f8380c0b11576057e26e098c6d3b3"
server_key_iv: "6b100b5e016d15c6516de020c54e0b75"
server_key_version: "encryptionkey@0"
I20260927 02:56:41.354741 21877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:41.355444 21892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:41.355592 21877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:41.355659 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "f6f15d706dec4be2bed71671cec2b974"
format_stamp: "Formatted at 2026-09-27 02:56:41 on dist-test-slave-mfx9"
server_key: "352f8380c0b11576057e26e098c6d3b3"
server_key_iv: "6b100b5e016d15c6516de020c54e0b75"
server_key_version: "encryptionkey@0"
I20260927 02:56:41.355742 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:41.365811 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:41.367441 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:41.367551 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:41.371011 21877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46075
I20260927 02:56:41.371042 21944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46075 every 8 connection(s)
I20260927 02:56:41.371326 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:41.372074 21945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:41.374107 21945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974: Bootstrap starting.
I20260927 02:56:41.374590 21945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:41.374883 21945 log.cc:824] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:41.375470 21945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974: No bootstrap required, opened a new log
I20260927 02:56:41.376878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21877
I20260927 02:56:41.376965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:41.376914 21945 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f15d706dec4be2bed71671cec2b974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46075 } }
I20260927 02:56:41.377065 21945 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:41.377110 392 external_mini_cluster.cc:1428] Setting key 1f05a9aaea9b3f5c2f540ccab2ecf999
I20260927 02:56:41.377108 21945 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f15d706dec4be2bed71671cec2b974, State: Initialized, Role: FOLLOWER
I20260927 02:56:41.377245 21945 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f15d706dec4be2bed71671cec2b974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46075 } }
I20260927 02:56:41.377323 21945 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:41.377353 21945 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:41.377398 21945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:41.378331 21945 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f15d706dec4be2bed71671cec2b974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46075 } }
I20260927 02:56:41.378441 21945 leader_election.cc:302] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f15d706dec4be2bed71671cec2b974; no voters:
I20260927 02:56:41.378599 21945 leader_election.cc:288] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:41.378620 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:41.378739 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [term 1 LEADER]: Becoming Leader. State: Replica: f6f15d706dec4be2bed71671cec2b974, State: Running, Role: LEADER
I20260927 02:56:41.378850 21945 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:41.378901 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f15d706dec4be2bed71671cec2b974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46075 } }
I20260927 02:56:41.379261 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f15d706dec4be2bed71671cec2b974. Latest consensus state: current_term: 1 leader_uuid: "f6f15d706dec4be2bed71671cec2b974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f15d706dec4be2bed71671cec2b974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46075 } } }
I20260927 02:56:41.379339 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:41.379534 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f15d706dec4be2bed71671cec2b974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f15d706dec4be2bed71671cec2b974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46075 } } }
I20260927 02:56:41.379777 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974 [sys.catalog]: This master's current role is: LEADER
W20260927 02:56:41.380527 21966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:41.381109 21966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:41.381191 21956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:41.381366 21956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:41.383131 21956 catalog_manager.cc:1347] Generated new cluster ID: 13c3baf02a2042438e38ebc3e287e138
I20260927 02:56:41.383173 21956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:56:41 dist-test-slave-mfx9 krb5kdc[21864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:41.388042 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:41.377489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53743 (local address 127.0.98.62:46075)
0927 02:56:41.377650 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:41.377657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:41.377772 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:56:41.377910 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:41.377915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:41.377954 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:41.378056 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:41.378804 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:41.380088 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:41.380865 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:41.381477 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:41.381582 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:41.384235 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:41.384239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:41.384239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:41.384249 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:41.386419 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:41.386870 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:41.386873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:41.386873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:41.386980 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:41.387198 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:41.387199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:41.387199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:41.387475 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:41.387603 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:41.387736 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:56:41.387826 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260927 02:56:41.388955 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:41.397276 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:41.390283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49517 (local address 127.0.98.62:46075)
0927 02:56:41.390323 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:41.390324 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:41.390427 (+ 103us) server_negotiation.cc:407] Connection header received
0927 02:56:41.390455 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:41.390456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:41.390465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:41.390520 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:41.391275 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:41.392059 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:41.393063 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:41.393383 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:41.393419 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:41.394735 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:41.394736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:41.394736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:41.394742 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:41.396090 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:41.396478 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:41.396479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:41.396479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:41.396527 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:41.396741 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:41.396741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:41.396741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:41.396848 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:41.396916 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:41.397136 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 02:56:41.397195 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:56:41.398378 21956 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:41.399096 21956 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:41.401592 21956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f15d706dec4be2bed71671cec2b974: Generated new TSK 0
I20260927 02:56:41.401757 21956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:41.406411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21877
W20260927 02:56:41.409761 21862 connection.cc:535] client connection to 127.0.98.62:46075 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46075): Connection reset by peer (error 104)
2026-09-27T02:56:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:41 dist-test-slave-mfx9 krb5kdc[21976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:41 dist-test-slave-mfx9 krb5kdc[21976](info): set up 2 sockets
Sep 27 02:56:41 dist-test-slave-mfx9 krb5kdc[21976](info): commencing operation
krb5kdc: starting...
W20260927 02:56:42.346477 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:42 dist-test-slave-mfx9 krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:42.451606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36159
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:42.527572 21989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:42.527766 21989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:42.527796 21989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:42.528893 21989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:42.528930 21989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:42.528949 21989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:42.528964 21989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:42.528980 21989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:42.530349 21989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:42.530571 21989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:42.530783 21989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:42.535089 21997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:42.535215 21989 server_base.cc:1034] running on GCE node
W20260927 02:56:42.535106 21994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:42.535100 21995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:42.535547 21989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:42.535840 21989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:42.536985 21989 hybrid_clock.cc:648] HybridClock initialized: now 1790477802536969 us; error 26 us; skew 500 ppm
Sep 27 02:56:42 dist-test-slave-mfx9 krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:42.539196 21989 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:42.539577 21989 webserver.cc:460] Webserver started at http://127.0.98.62:46421/ using document root <none> and password file <none>
I20260927 02:56:42.539795 21989 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:42.539842 21989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:42.539960 21989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:42.540935 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "f57b9b2450af4b39810edef3f4bd8985"
format_stamp: "Formatted at 2026-09-27 02:56:42 on dist-test-slave-mfx9"
server_key: "9ae20ee8ed801b6f9ffb9fb26f81fa75"
server_key_iv: "16020c94d0ab0c35e3b84ca4eedf9905"
server_key_version: "encryptionkey@0"
I20260927 02:56:42.541244 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "f57b9b2450af4b39810edef3f4bd8985"
format_stamp: "Formatted at 2026-09-27 02:56:42 on dist-test-slave-mfx9"
server_key: "9ae20ee8ed801b6f9ffb9fb26f81fa75"
server_key_iv: "16020c94d0ab0c35e3b84ca4eedf9905"
server_key_version: "encryptionkey@0"
I20260927 02:56:42.542349 21989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:56:42.542974 22004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:42.543115 21989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:42.543190 21989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "f57b9b2450af4b39810edef3f4bd8985"
format_stamp: "Formatted at 2026-09-27 02:56:42 on dist-test-slave-mfx9"
server_key: "9ae20ee8ed801b6f9ffb9fb26f81fa75"
server_key_iv: "16020c94d0ab0c35e3b84ca4eedf9905"
server_key_version: "encryptionkey@0"
I20260927 02:56:42.543251 21989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:42.550690 21989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:42.552245 21989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:42.552362 21989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:42.555874 21989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36159
I20260927 02:56:42.555902 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36159 every 8 connection(s)
I20260927 02:56:42.556185 21989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:42.556896 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:42.559036 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985: Bootstrap starting.
I20260927 02:56:42.559645 22057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:42.559902 22057 log.cc:824] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:42.560575 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985: No bootstrap required, opened a new log
I20260927 02:56:42.562053 22057 raft_consensus.cc:348] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57b9b2450af4b39810edef3f4bd8985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } }
I20260927 02:56:42.562180 22057 raft_consensus.cc:374] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:42.562201 22057 raft_consensus.cc:729] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57b9b2450af4b39810edef3f4bd8985, State: Initialized, Role: FOLLOWER
I20260927 02:56:42.562321 22057 consensus_queue.cc:260] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57b9b2450af4b39810edef3f4bd8985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } }
I20260927 02:56:42.562384 22057 raft_consensus.cc:388] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:42.562415 22057 raft_consensus.cc:482] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:42.562459 22057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:42.562966 22057 raft_consensus.cc:504] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57b9b2450af4b39810edef3f4bd8985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } }
I20260927 02:56:42.563047 22057 leader_election.cc:302] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f57b9b2450af4b39810edef3f4bd8985; no voters:
I20260927 02:56:42.563175 22057 leader_election.cc:288] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:42.563227 22060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:42.563374 22060 raft_consensus.cc:686] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [term 1 LEADER]: Becoming Leader. State: Replica: f57b9b2450af4b39810edef3f4bd8985, State: Running, Role: LEADER
I20260927 02:56:42.563380 22057 sys_catalog.cc:564] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:42.563468 22060 consensus_queue.cc:237] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57b9b2450af4b39810edef3f4bd8985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } }
I20260927 02:56:42.563956 22061 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f57b9b2450af4b39810edef3f4bd8985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57b9b2450af4b39810edef3f4bd8985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } } }
I20260927 02:56:42.564039 22061 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:42.564118 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f57b9b2450af4b39810edef3f4bd8985. Latest consensus state: current_term: 1 leader_uuid: "f57b9b2450af4b39810edef3f4bd8985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57b9b2450af4b39810edef3f4bd8985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } } }
I20260927 02:56:42.564236 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:42.564483 22069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:42.564981 22069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:42.565198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 21989
I20260927 02:56:42.565301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:42.565459 392 external_mini_cluster.cc:1428] Setting key b0c824c2c7aa3145b5d1b59845abd05f
I20260927 02:56:42.566322 22069 catalog_manager.cc:1347] Generated new cluster ID: 88635fec191d42449b8bd8a99f213105
I20260927 02:56:42.566371 22069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:42.571153 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:42.565823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36699 (local address 127.0.98.62:36159)
0927 02:56:42.566131 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:42.566140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:42.566172 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:56:42.566280 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:42.566287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:42.566339 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:42.566479 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:42.566681 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:42.568068 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:42.569090 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:42.569735 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:42.569843 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:42.570482 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:42.570486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:42.570486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:42.570696 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:42.570734 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:42.570749 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:42.570853 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:42.570923 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:56:42.570949 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":77,"threads_started":1}
I20260927 02:56:42.572023 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:42.577041 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:42.573447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60259 (local address 127.0.98.62:36159)
0927 02:56:42.573480 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:42.573482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:42.573585 (+ 103us) server_negotiation.cc:407] Connection header received
0927 02:56:42.573686 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:42.573687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:42.573706 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:42.573735 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:42.573912 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:42.574662 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:42.575708 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:42.576016 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:42.576053 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:42.576788 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:42.576789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:42.576789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:42.576793 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:42.576806 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:42.576810 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:42.576840 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:42.576957 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:56:42.576969 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:56:42.587954 22069 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:42.588634 22069 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:42.591589 22069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f57b9b2450af4b39810edef3f4bd8985: Generated new TSK 0
I20260927 02:56:42.591758 22069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:42.594468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 21989
W20260927 02:56:42.598067 21974 connection.cc:535] client connection to 127.0.98.62:36159 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36159): Connection reset by peer (error 104)
2026-09-27T02:56:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:42 dist-test-slave-mfx9 krb5kdc[22088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:42 dist-test-slave-mfx9 krb5kdc[22088](info): set up 2 sockets
Sep 27 02:56:42 dist-test-slave-mfx9 krb5kdc[22088](info): commencing operation
krb5kdc: starting...
W20260927 02:56:43.542481 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:43 dist-test-slave-mfx9 krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:43.648947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42093
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:43.725435 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:43.725613 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:43.725642 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:43.726739 22101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:43.726795 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:43.726809 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:43.726821 22101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:43.726837 22101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:43.728217 22101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:43.728387 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:43.728534 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:43.733058 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:43.733070 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:43.733141 22101 server_base.cc:1034] running on GCE node
W20260927 02:56:43.733081 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:43.733510 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:43.733776 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:43.734905 22101 hybrid_clock.cc:648] HybridClock initialized: now 1790477803734889 us; error 33 us; skew 500 ppm
Sep 27 02:56:43 dist-test-slave-mfx9 krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:43.737188 22101 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:43.737555 22101 webserver.cc:460] Webserver started at http://127.0.98.62:36757/ using document root <none> and password file <none>
I20260927 02:56:43.737733 22101 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:43.737771 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:43.737859 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:43.738814 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "e53cf1ef248145c2bbaebd9d2c8f030e"
format_stamp: "Formatted at 2026-09-27 02:56:43 on dist-test-slave-mfx9"
server_key: "12c71c2c89fd064b4de016543cc85901"
server_key_iv: "c763c824ae6527c32160585e2ebd31f9"
server_key_version: "encryptionkey@0"
I20260927 02:56:43.739110 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "e53cf1ef248145c2bbaebd9d2c8f030e"
format_stamp: "Formatted at 2026-09-27 02:56:43 on dist-test-slave-mfx9"
server_key: "12c71c2c89fd064b4de016543cc85901"
server_key_iv: "c763c824ae6527c32160585e2ebd31f9"
server_key_version: "encryptionkey@0"
I20260927 02:56:43.740211 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:56:43.740826 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:43.740978 22101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:43.741058 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "e53cf1ef248145c2bbaebd9d2c8f030e"
format_stamp: "Formatted at 2026-09-27 02:56:43 on dist-test-slave-mfx9"
server_key: "12c71c2c89fd064b4de016543cc85901"
server_key_iv: "c763c824ae6527c32160585e2ebd31f9"
server_key_version: "encryptionkey@0"
I20260927 02:56:43.741117 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:43.744185 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:43.745738 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:43.745857 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:43.749239 22101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42093
I20260927 02:56:43.749270 22168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42093 every 8 connection(s)
I20260927 02:56:43.749526 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:43.750391 22169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:43.752317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 22101
I20260927 02:56:43.752424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:43.752444 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e: Bootstrap starting.
I20260927 02:56:43.752580 392 external_mini_cluster.cc:1428] Setting key 38ed3606a3d72c6167ca3c7e16e2732b
I20260927 02:56:43.753047 22169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:43.753332 22169 log.cc:824] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:43.753954 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e: No bootstrap required, opened a new log
I20260927 02:56:43.754803 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:43.752983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60047 (local address 127.0.98.62:42093)
0927 02:56:43.753221 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:43.753228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:43.753251 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:56:43.753678 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:43.753684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:43.753710 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:43.753821 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:43.754420 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:43.754423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:43.754423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:43.754426 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:43.754456 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:43.754465 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:43.754518 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:43.754578 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 02:56:43.754606 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":63,"threads_started":1}
I20260927 02:56:43.755607 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:43.756383 22169 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53cf1ef248145c2bbaebd9d2c8f030e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } }
I20260927 02:56:43.756559 22169 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:43.756616 22169 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53cf1ef248145c2bbaebd9d2c8f030e, State: Initialized, Role: FOLLOWER
I20260927 02:56:43.756729 22169 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53cf1ef248145c2bbaebd9d2c8f030e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } }
I20260927 02:56:43.756808 22169 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:43.756839 22169 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:43.756910 22169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:43.757380 22169 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53cf1ef248145c2bbaebd9d2c8f030e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } }
I20260927 02:56:43.757462 22169 leader_election.cc:302] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53cf1ef248145c2bbaebd9d2c8f030e; no voters:
I20260927 02:56:43.757608 22169 leader_election.cc:288] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:43.757642 22179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:43.757771 22179 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [term 1 LEADER]: Becoming Leader. State: Replica: e53cf1ef248145c2bbaebd9d2c8f030e, State: Running, Role: LEADER
I20260927 02:56:43.757807 22169 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:43.757854 22179 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53cf1ef248145c2bbaebd9d2c8f030e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } }
I20260927 02:56:43.758160 22180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53cf1ef248145c2bbaebd9d2c8f030e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53cf1ef248145c2bbaebd9d2c8f030e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } } }
I20260927 02:56:43.758172 22181 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53cf1ef248145c2bbaebd9d2c8f030e. Latest consensus state: current_term: 1 leader_uuid: "e53cf1ef248145c2bbaebd9d2c8f030e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53cf1ef248145c2bbaebd9d2c8f030e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } } }
I20260927 02:56:43.758253 22180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:43.758260 22181 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:43.758813 22187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:43.759253 22187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:43.759716 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:43.756850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38467 (local address 127.0.98.62:42093)
0927 02:56:43.756881 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:43.756883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:43.758909 (+ 2026us) server_negotiation.cc:407] Connection header received
0927 02:56:43.759000 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:43.759001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:43.759005 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:43.759045 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:43.759501 (+ 456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:43.759502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:43.759502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:43.759506 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:43.759518 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:43.759523 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:43.759567 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:43.759650 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 02:56:43.759662 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:56:43.763125 22187 catalog_manager.cc:1347] Generated new cluster ID: 8a83ef009ff34f1c875db485723e4a89
I20260927 02:56:43.763165 22187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:43.775344 22187 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:43.775471 22187 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:43.781893 22187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53cf1ef248145c2bbaebd9d2c8f030e: Generated new TSK 0
I20260927 02:56:43.782029 22187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:43.788457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 22101
2026-09-27T02:56:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:43 dist-test-slave-mfx9 krb5kdc[22200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:43 dist-test-slave-mfx9 krb5kdc[22200](info): set up 2 sockets
Sep 27 02:56:43 dist-test-slave-mfx9 krb5kdc[22200](info): commencing operation
krb5kdc: starting...
W20260927 02:56:44.726555 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:44 dist-test-slave-mfx9 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:44.835235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:44.912532 22213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:44.912717 22213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:44.912738 22213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:44.913780 22213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:44.913813 22213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:44.913826 22213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:44.913837 22213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:44.913847 22213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:44.915249 22213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:44.915423 22213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:44.915572 22213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:44.919888 22218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:44.919986 22219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:44.920050 22213 server_base.cc:1034] running on GCE node
W20260927 02:56:44.919947 22221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:44.920310 22213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:44.920553 22213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:44.921685 22213 hybrid_clock.cc:648] HybridClock initialized: now 1790477804921672 us; error 26 us; skew 500 ppm
Sep 27 02:56:44 dist-test-slave-mfx9 krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:44.923940 22213 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:44.924322 22213 webserver.cc:460] Webserver started at http://127.0.98.62:44399/ using document root <none> and password file <none>
I20260927 02:56:44.924515 22213 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:44.924561 22213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:44.924676 22213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:44.925655 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "d04ad73644ab4cbca0ba0af2d73616e9"
format_stamp: "Formatted at 2026-09-27 02:56:44 on dist-test-slave-mfx9"
server_key: "f3f9676afa7012f574f531f6502ea498"
server_key_iv: "74a87b1510b674ec3d74193c1b882aad"
server_key_version: "encryptionkey@0"
I20260927 02:56:44.925966 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "d04ad73644ab4cbca0ba0af2d73616e9"
format_stamp: "Formatted at 2026-09-27 02:56:44 on dist-test-slave-mfx9"
server_key: "f3f9676afa7012f574f531f6502ea498"
server_key_iv: "74a87b1510b674ec3d74193c1b882aad"
server_key_version: "encryptionkey@0"
I20260927 02:56:44.927183 22213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:44.927838 22228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:44.927991 22213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:44.928061 22213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "d04ad73644ab4cbca0ba0af2d73616e9"
format_stamp: "Formatted at 2026-09-27 02:56:44 on dist-test-slave-mfx9"
server_key: "f3f9676afa7012f574f531f6502ea498"
server_key_iv: "74a87b1510b674ec3d74193c1b882aad"
server_key_version: "encryptionkey@0"
I20260927 02:56:44.928117 22213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:44.941766 22213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:44.943483 22213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:44.943609 22213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:44.947137 22213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35831
I20260927 02:56:44.947170 22280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35831 every 8 connection(s)
I20260927 02:56:44.947485 22213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:44.948264 22281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:44.949453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 22213
I20260927 02:56:44.949548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:44.949700 392 external_mini_cluster.cc:1428] Setting key d9d34d40d05a38df5edf1bdc7a048eb2
I20260927 02:56:44.950626 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9: Bootstrap starting.
I20260927 02:56:44.951277 22281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:44.951591 22281 log.cc:824] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:44.952245 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9: No bootstrap required, opened a new log
I20260927 02:56:44.954348 22281 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ad73644ab4cbca0ba0af2d73616e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35831 } }
I20260927 02:56:44.954496 22281 raft_consensus.cc:374] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:44.954519 22281 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04ad73644ab4cbca0ba0af2d73616e9, State: Initialized, Role: FOLLOWER
I20260927 02:56:44.954632 22281 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ad73644ab4cbca0ba0af2d73616e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35831 } }
I20260927 02:56:44.954713 22281 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:44.954785 22281 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:44.954838 22281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:44.955485 22281 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ad73644ab4cbca0ba0af2d73616e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35831 } }
I20260927 02:56:44.955596 22281 leader_election.cc:302] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04ad73644ab4cbca0ba0af2d73616e9; no voters:
I20260927 02:56:44.955765 22281 leader_election.cc:288] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:44.955813 22286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:44.955994 22286 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [term 1 LEADER]: Becoming Leader. State: Replica: d04ad73644ab4cbca0ba0af2d73616e9, State: Running, Role: LEADER
I20260927 02:56:44.956012 22281 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:44.956115 22286 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ad73644ab4cbca0ba0af2d73616e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35831 } }
Sep 27 02:56:44 dist-test-slave-mfx9 krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:44.957369 22287 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d04ad73644ab4cbca0ba0af2d73616e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ad73644ab4cbca0ba0af2d73616e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35831 } } }
I20260927 02:56:44.957490 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04ad73644ab4cbca0ba0af2d73616e9. Latest consensus state: current_term: 1 leader_uuid: "d04ad73644ab4cbca0ba0af2d73616e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ad73644ab4cbca0ba0af2d73616e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35831 } } }
I20260927 02:56:44.957617 22287 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:44.957624 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9 [sys.catalog]: This master's current role is: LEADER
W20260927 02:56:44.958081 22301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:44.958124 22301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:44.958210 22302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:44.958379 22302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:44.960045 22302 catalog_manager.cc:1347] Generated new cluster ID: faf0f44d5f2c4fd1b9fa97633fbcce74
I20260927 02:56:44.960090 22302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:44.960304 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:44.950163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35183 (local address 127.0.98.62:35831)
0927 02:56:44.950334 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:44.950341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:44.950365 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:56:44.950787 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:44.950795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:44.950853 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:44.950963 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:44.951734 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:44.952651 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:44.953665 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:44.954048 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:44.954119 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:44.956861 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:44.956866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:44.956866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:44.956878 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:44.958806 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:44.959134 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:44.959137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:44.959137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:44.959245 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:44.959450 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:44.959451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:44.959451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:44.959726 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:44.959872 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:44.959992 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 02:56:44.960122 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260927 02:56:44.961059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:44.969235 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:44.962504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57295 (local address 127.0.98.62:35831)
0927 02:56:44.962542 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:44.962544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:44.962550 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:56:44.962566 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:44.962567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:44.962574 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:44.962611 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:44.963441 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:44.964198 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:44.965136 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:44.965433 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:44.965468 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:44.966817 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:44.966818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:44.966818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:44.966824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:44.968202 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:44.968525 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:44.968525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:44.968526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:44.968567 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:44.968720 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:44.968720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:44.968720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:44.968837 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:44.968899 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:44.969071 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 02:56:44.969143 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:56:44.970736 22302 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:44.971251 22302 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:44.979102 22302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d04ad73644ab4cbca0ba0af2d73616e9: Generated new TSK 0
I20260927 02:56:44.979272 22302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:44.984305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 22213
2026-09-27T02:56:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:45 dist-test-slave-mfx9 krb5kdc[22312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:45 dist-test-slave-mfx9 krb5kdc[22312](info): set up 2 sockets
Sep 27 02:56:45 dist-test-slave-mfx9 krb5kdc[22312](info): commencing operation
krb5kdc: starting...
W20260927 02:56:45.934482 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:46 dist-test-slave-mfx9 krb5kdc[22312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:46.041043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:46.117341 22325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:46.117538 22325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:46.117558 22325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:46.118661 22325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:46.118701 22325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:46.118713 22325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:46.118726 22325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:46.118736 22325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:46.120162 22325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:46.120386 22325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:46.120584 22325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:46.125074 22331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:46.125100 22333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:46.125174 22325 server_base.cc:1034] running on GCE node
W20260927 02:56:46.125272 22330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:46.125506 22325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:46.125766 22325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:46.126899 22325 hybrid_clock.cc:648] HybridClock initialized: now 1790477806126884 us; error 28 us; skew 500 ppm
Sep 27 02:56:46 dist-test-slave-mfx9 krb5kdc[22312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:46.129153 22325 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:46.129532 22325 webserver.cc:460] Webserver started at http://127.0.98.62:33553/ using document root <none> and password file <none>
I20260927 02:56:46.129720 22325 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:46.129765 22325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:46.129884 22325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:46.130988 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "e1d45b449dee4ae587b8262af832eccf"
format_stamp: "Formatted at 2026-09-27 02:56:46 on dist-test-slave-mfx9"
server_key: "494b34c64e85a41100ed895b64e48119"
server_key_iv: "8ee2fcd502ff86f7527c6af2464a19e9"
server_key_version: "encryptionkey@0"
I20260927 02:56:46.131314 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "e1d45b449dee4ae587b8262af832eccf"
format_stamp: "Formatted at 2026-09-27 02:56:46 on dist-test-slave-mfx9"
server_key: "494b34c64e85a41100ed895b64e48119"
server_key_iv: "8ee2fcd502ff86f7527c6af2464a19e9"
server_key_version: "encryptionkey@0"
I20260927 02:56:46.132512 22325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:56:46.133160 22340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:46.133303 22325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:46.133385 22325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "e1d45b449dee4ae587b8262af832eccf"
format_stamp: "Formatted at 2026-09-27 02:56:46 on dist-test-slave-mfx9"
server_key: "494b34c64e85a41100ed895b64e48119"
server_key_iv: "8ee2fcd502ff86f7527c6af2464a19e9"
server_key_version: "encryptionkey@0"
I20260927 02:56:46.133442 22325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:46.144016 22325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:46.145979 22325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:46.146116 22325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:46.149597 22325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34837
I20260927 02:56:46.149629 22392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34837 every 8 connection(s)
I20260927 02:56:46.149920 22325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:46.150673 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:46.152753 22393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf: Bootstrap starting.
I20260927 02:56:46.153258 22393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:46.153497 22393 log.cc:824] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:46.154083 22393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf: No bootstrap required, opened a new log
I20260927 02:56:46.154780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 22325
I20260927 02:56:46.154883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:46.155037 392 external_mini_cluster.cc:1428] Setting key 63611eec64af8e3b2ac7a3714eceab33
I20260927 02:56:46.155864 22393 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d45b449dee4ae587b8262af832eccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } }
I20260927 02:56:46.156004 22393 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:46.156038 22393 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d45b449dee4ae587b8262af832eccf, State: Initialized, Role: FOLLOWER
I20260927 02:56:46.156157 22393 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d45b449dee4ae587b8262af832eccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } }
I20260927 02:56:46.156244 22393 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:46.156275 22393 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:46.156322 22393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:46.156796 22393 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d45b449dee4ae587b8262af832eccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } }
I20260927 02:56:46.156875 22393 leader_election.cc:302] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1d45b449dee4ae587b8262af832eccf; no voters:
I20260927 02:56:46.157020 22393 leader_election.cc:288] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:46.157047 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:46.157191 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [term 1 LEADER]: Becoming Leader. State: Replica: e1d45b449dee4ae587b8262af832eccf, State: Running, Role: LEADER
I20260927 02:56:46.157267 22393 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:46.157323 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d45b449dee4ae587b8262af832eccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } }
I20260927 02:56:46.157653 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1d45b449dee4ae587b8262af832eccf. Latest consensus state: current_term: 1 leader_uuid: "e1d45b449dee4ae587b8262af832eccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d45b449dee4ae587b8262af832eccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } }
I20260927 02:56:46.157873 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:46.157811 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1d45b449dee4ae587b8262af832eccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d45b449dee4ae587b8262af832eccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } }
I20260927 02:56:46.158227 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:46.158424 22407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:46.158957 22407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:46.160291 22407 catalog_manager.cc:1347] Generated new cluster ID: 27493bb17b374673ad45261cf576f312
I20260927 02:56:46.160337 22407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:46.160854 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:46.155506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51797 (local address 127.0.98.62:34837)
0927 02:56:46.155659 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:46.155667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:46.155698 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:56:46.155824 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:46.155829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:46.155877 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:46.155977 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:46.156208 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:46.157575 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:46.158713 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:46.159308 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:46.159419 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:46.160042 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:46.160046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:46.160046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:46.160256 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:46.160291 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:46.160305 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:46.160550 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:46.160583 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 02:56:46.160645 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260927 02:56:46.161729 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:46.169938 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:46.163063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54051 (local address 127.0.98.62:34837)
0927 02:56:46.163092 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:46.163093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:46.166924 (+ 3831us) server_negotiation.cc:407] Connection header received
0927 02:56:46.166986 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:46.166987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:46.166997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:46.167034 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:46.167155 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:46.167851 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:46.168815 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:46.169082 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:46.169132 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:46.169725 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:46.169726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:46.169727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:46.169730 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:46.169743 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:46.169747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:46.169786 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:46.169847 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:56:46.169867 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:56:46.171571 22407 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:46.172029 22407 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:46.184522 22407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1d45b449dee4ae587b8262af832eccf: Generated new TSK 0
I20260927 02:56:46.184681 22407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:46.192842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 22325
2026-09-27T02:56:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:46 dist-test-slave-mfx9 krb5kdc[22424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:46 dist-test-slave-mfx9 krb5kdc[22424](info): set up 2 sockets
Sep 27 02:56:46 dist-test-slave-mfx9 krb5kdc[22424](info): commencing operation
krb5kdc: starting...
W20260927 02:56:47.130869 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:47 dist-test-slave-mfx9 krb5kdc[22424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:47.240898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:47.317204 22437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:47.317389 22437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:47.317418 22437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:47.318511 22437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:47.318549 22437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:47.318567 22437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:47.318583 22437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:47.318599 22437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:47.320047 22437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:47.320268 22437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:47.320472 22437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:47.324875 22442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:47.324884 22445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:47.324884 22443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:47.324982 22437 server_base.cc:1034] running on GCE node
I20260927 02:56:47.325289 22437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:47.325537 22437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:47.326680 22437 hybrid_clock.cc:648] HybridClock initialized: now 1790477807326666 us; error 33 us; skew 500 ppm
Sep 27 02:56:47 dist-test-slave-mfx9 krb5kdc[22424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:47.328929 22437 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:47.329276 22437 webserver.cc:460] Webserver started at http://127.0.98.62:33893/ using document root <none> and password file <none>
I20260927 02:56:47.329442 22437 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:47.329479 22437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:47.329562 22437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:47.330484 22437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "2dc1deb815a1418e9e32a6c1b0c6c286"
format_stamp: "Formatted at 2026-09-27 02:56:47 on dist-test-slave-mfx9"
server_key: "b0b6242ba1a81c7bd62d38c1c2f2963b"
server_key_iv: "bdb71259aa3e120ce7d999b633d96c14"
server_key_version: "encryptionkey@0"
I20260927 02:56:47.330792 22437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "2dc1deb815a1418e9e32a6c1b0c6c286"
format_stamp: "Formatted at 2026-09-27 02:56:47 on dist-test-slave-mfx9"
server_key: "b0b6242ba1a81c7bd62d38c1c2f2963b"
server_key_iv: "bdb71259aa3e120ce7d999b633d96c14"
server_key_version: "encryptionkey@0"
I20260927 02:56:47.331959 22437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:56:47.332597 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:47.332782 22437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:47.332862 22437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "2dc1deb815a1418e9e32a6c1b0c6c286"
format_stamp: "Formatted at 2026-09-27 02:56:47 on dist-test-slave-mfx9"
server_key: "b0b6242ba1a81c7bd62d38c1c2f2963b"
server_key_iv: "bdb71259aa3e120ce7d999b633d96c14"
server_key_version: "encryptionkey@0"
I20260927 02:56:47.332916 22437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:47.335984 22437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:47.337837 22437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:47.337931 22437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:47.341320 22437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42451
I20260927 02:56:47.341351 22504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42451 every 8 connection(s)
I20260927 02:56:47.341603 22437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:47.342463 22505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:47.344558 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286: Bootstrap starting.
I20260927 02:56:47.344805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 22437
I20260927 02:56:47.344890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:47.345033 392 external_mini_cluster.cc:1428] Setting key 9a9c0e018b823651fc0712ebe8d8bc11
I20260927 02:56:47.345127 22505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:47.345440 22505 log.cc:824] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:47.346091 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286: No bootstrap required, opened a new log
I20260927 02:56:47.347232 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:47.345457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52081 (local address 127.0.98.62:42451)
0927 02:56:47.345640 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:47.345650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:47.345683 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:56:47.346131 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:47.346140 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:47.346173 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:47.346289 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:47.346856 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:47.346858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:47.346859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:47.346862 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:47.346897 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:47.346906 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:47.346967 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:47.347028 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:56:47.347059 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260927 02:56:47.348106 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:47.348315 22505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc1deb815a1418e9e32a6c1b0c6c286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:56:47.348440 22505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:56:47.348462 22505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc1deb815a1418e9e32a6c1b0c6c286, State: Initialized, Role: FOLLOWER
I20260927 02:56:47.348569 22505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc1deb815a1418e9e32a6c1b0c6c286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:56:47.348647 22505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:56:47.348681 22505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:56:47.348727 22505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:47.349268 22505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc1deb815a1418e9e32a6c1b0c6c286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:56:47.349368 22505 leader_election.cc:302] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dc1deb815a1418e9e32a6c1b0c6c286; no voters:
I20260927 02:56:47.349511 22505 leader_election.cc:288] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:56:47.349545 22515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:47.349670 22515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [term 1 LEADER]: Becoming Leader. State: Replica: 2dc1deb815a1418e9e32a6c1b0c6c286, State: Running, Role: LEADER
I20260927 02:56:47.349730 22505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:47.349758 22515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc1deb815a1418e9e32a6c1b0c6c286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:56:47.350363 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dc1deb815a1418e9e32a6c1b0c6c286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc1deb815a1418e9e32a6c1b0c6c286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260927 02:56:47.350481 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:47.350566 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:47.349419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55213 (local address 127.0.98.62:42451)
0927 02:56:47.349471 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:47.349472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:47.349582 (+ 110us) server_negotiation.cc:407] Connection header received
0927 02:56:47.349665 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:47.349666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:47.349671 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:47.349725 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:47.350251 (+ 526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:47.350252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:47.350252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:56:47.350256 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:47.350269 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:56:47.350273 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:47.350326 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:47.350503 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 02:56:47.350514 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:56:47.351080 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dc1deb815a1418e9e32a6c1b0c6c286. Latest consensus state: current_term: 1 leader_uuid: "2dc1deb815a1418e9e32a6c1b0c6c286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc1deb815a1418e9e32a6c1b0c6c286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260927 02:56:47.351155 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286 [sys.catalog]: This master's current role is: LEADER
W20260927 02:56:47.351260 22530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:47.351310 22530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:47.351362 22525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:47.351490 22525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:47.352654 22525 catalog_manager.cc:1347] Generated new cluster ID: 168bbad3b846433cbc6d9d927528e191
I20260927 02:56:47.352705 22525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:47.363867 22525 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:47.364010 22525 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:47.366777 22525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dc1deb815a1418e9e32a6c1b0c6c286: Generated new TSK 0
I20260927 02:56:47.366928 22525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:47.374101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 22437
2026-09-27T02:56:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1181 ms)
[----------] 18 tests from AuthTokenIssuingTest (21614 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:47 dist-test-slave-mfx9 krb5kdc[22536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:47 dist-test-slave-mfx9 krb5kdc[22536](info): set up 2 sockets
Sep 27 02:56:47 dist-test-slave-mfx9 krb5kdc[22536](info): commencing operation
krb5kdc: starting...
W20260927 02:56:48.318495 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:48.423599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40009
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40009,127.0.98.61:32995,127.0.98.60:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:48.498991 22549 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:56:48.499174 22549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:48.499189 22549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:48.499207 22549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:48.500298 22549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:48.500334 22549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:48.500345 22549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:48.500370 22549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:48.500383 22549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:48.501799 22549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40009,127.0.98.61:32995,127.0.98.60:34161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:48.502143 22549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:48.502347 22549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:48.506666 22554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:48.506742 22555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:48.506666 22557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:48.507005 22549 server_base.cc:1034] running on GCE node
I20260927 02:56:48.507236 22549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:48.507494 22549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:48.508635 22549 hybrid_clock.cc:648] HybridClock initialized: now 1790477808508617 us; error 29 us; skew 500 ppm
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:48.511157 22549 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:48.511519 22549 webserver.cc:460] Webserver started at http://127.0.98.62:41059/ using document root <none> and password file <none>
I20260927 02:56:48.511682 22549 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:48.511718 22549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:48.511804 22549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:48.512755 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "4b5afbc772e443ef8648b07ab93c9867"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "de5db6985164ee0d52ccbc2fd79919f0"
server_key_iv: "0acf62d093edfd06a67575f72eab6c20"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.513038 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b5afbc772e443ef8648b07ab93c9867"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "de5db6985164ee0d52ccbc2fd79919f0"
server_key_iv: "0acf62d093edfd06a67575f72eab6c20"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.514185 22549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:56:48.514791 22564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:48.514932 22549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:48.515009 22549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "4b5afbc772e443ef8648b07ab93c9867"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "de5db6985164ee0d52ccbc2fd79919f0"
server_key_iv: "0acf62d093edfd06a67575f72eab6c20"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.515061 22549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:48.530385 22549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:48.532322 22549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:48.532423 22549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:48.535881 22549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40009
I20260927 02:56:48.535928 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40009 every 8 connection(s)
I20260927 02:56:48.536206 22549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:48.537135 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:48.537288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 22549
I20260927 02:56:48.537389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:48.537545 392 external_mini_cluster.cc:1428] Setting key f4779cb27b4ec42778e69605fdb333da
I20260927 02:56:48.538710 22617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:48.550364 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.539447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.0.98.62:40009)
0927 02:56:48.539577 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.539586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.539703 (+ 117us) server_negotiation.cc:407] Connection header received
0927 02:56:48.540176 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.540179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.540199 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.540295 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.540776 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.542038 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.542881 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.543359 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.543470 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.547343 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.547351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.547351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.547358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.548743 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.549171 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.549174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.549175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.549300 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.549534 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.549535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.549535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.549778 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.549858 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.550069 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.550170 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20260927 02:56:48.550371 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.539398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40009 (local address 127.0.0.1:55510)
0927 02:56:48.539585 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.539618 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.540037 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.540307 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.540309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.540543 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.540688 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.540691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.542123 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.542124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.542805 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.542808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.543450 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.543451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.543499 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.543903 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.543911 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.547266 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.548828 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.548834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.548836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.549084 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.549383 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.549383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.549384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.549472 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.549863 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.549870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.549946 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.550048 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.550230 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":49,"thread_start_us":79,"threads_started":1}
I20260927 02:56:48.551097 22617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:48.551379 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.551288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32995 (local address 127.0.0.1:41634)
0927 02:56:48.551328 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.551364 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:56:48.551496 22568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32995: connect: Connection refused (error 111)
I20260927 02:56:48.551853 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.551779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32995 (local address 127.0.0.1:41650)
0927 02:56:48.551817 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.551839 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:56:48.551944 22617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32995: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32995: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:56:48.559661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32995
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40009,127.0.98.61:32995,127.0.98.60:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:56:48.601176 22617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } attempt: 1
I20260927 02:56:48.601709 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.601553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32995 (local address 127.0.0.1:41666)
0927 02:56:48.601612 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.601676 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 02:56:48.602090 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.601988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32995 (local address 127.0.0.1:41670)
0927 02:56:48.602034 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.602064 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:56:48.602195 22617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32995: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32995: connect: Connection refused (error 111)
W20260927 02:56:48.636322 22624 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:56:48.636514 22624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:48.636536 22624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:48.636559 22624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:48.637704 22624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:48.637743 22624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:48.637761 22624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:48.637776 22624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:48.637791 22624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:48.639232 22624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40009,127.0.98.61:32995,127.0.98.60:34161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:48.639444 22624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:48.639633 22624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:48.644011 22632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:48.644062 22624 server_base.cc:1034] running on GCE node
W20260927 02:56:48.644030 22630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:48.644038 22629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:48.644444 22624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:48.644726 22624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:48.645867 22624 hybrid_clock.cc:648] HybridClock initialized: now 1790477808645857 us; error 23 us; skew 500 ppm
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:48.648101 22624 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:56:48.648494 22624 webserver.cc:460] Webserver started at http://127.0.98.61:34575/ using document root <none> and password file <none>
I20260927 02:56:48.648685 22624 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:48.648746 22624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:48.648864 22624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:48.649806 22624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/data/instance:
uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "b3afeb151b5fad7dca6fa99880d237cd"
server_key_iv: "a6be0261037d5ea0a80a57a63537bef1"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.650111 22624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/wal/instance:
uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "b3afeb151b5fad7dca6fa99880d237cd"
server_key_iv: "a6be0261037d5ea0a80a57a63537bef1"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.651321 22624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:48.651909 22639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:48.652050 22624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:48.652127 22624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/wal
uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "b3afeb151b5fad7dca6fa99880d237cd"
server_key_iv: "a6be0261037d5ea0a80a57a63537bef1"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.652184 22624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:48.665166 22624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:48.666805 22624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:48.666929 22624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:48.670329 22624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32995
I20260927 02:56:48.670322 22617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } attempt: 2
I20260927 02:56:48.670389 22691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32995 every 8 connection(s)
I20260927 02:56:48.670678 22624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:56:48.671344 22692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:48.673194 22692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:48.673447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 22624
I20260927 02:56:48.673533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-1/wal/instance
I20260927 02:56:48.673708 392 external_mini_cluster.cc:1428] Setting key 9985c13f31758757e04583b2aaf81de7
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:48.680035 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.670579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32995 (local address 127.0.0.1:41672)
0927 02:56:48.670641 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.670653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.670706 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.672025 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.672026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.672160 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.672202 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.672203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.673649 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.673650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.674419 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.674423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.675222 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.675223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.675292 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.675673 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.675677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.677065 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.678707 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.678708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.678708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.678928 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.679188 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.679189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.679189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.679255 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.679704 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.679706 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.679767 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.679829 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.679938 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:56:48.680516 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.670712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.0.98.61:32995)
0927 02:56:48.671286 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.671295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.671324 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:56:48.671834 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.671842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.671899 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.672037 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.672288 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.673577 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.674502 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.675135 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.675235 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.677148 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.677154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.677154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.677164 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.678636 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.679005 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.679008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.679008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.679106 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.679326 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.679327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.679327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.679601 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.679673 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.680182 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.680301 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":482,"threads_started":1}
I20260927 02:56:48.681228 22617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:48.681547 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.681438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35890)
0927 02:56:48.681491 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.681522 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 02:56:48.681774 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.674039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.0.98.62:40009)
0927 02:56:48.674073 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.674075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.674083 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:48.674100 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.674101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.674110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.674173 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.674631 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.675284 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.676274 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.676500 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.676549 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.678837 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.678838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.678838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.678845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.680038 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.680870 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.680871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.680871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.680917 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.681196 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.681197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.681197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.681294 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.681350 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.681620 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.681695 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:56:48.681842 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.681770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35892)
0927 02:56:48.681805 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.681826 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:56:48.681895 22617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34161: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
I20260927 02:56:48.682076 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.673820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40009 (local address 127.0.0.1:55520)
0927 02:56:48.673959 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.673999 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.674052 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.674190 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.674192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.674486 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.674563 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.674566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.675376 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.675377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.676211 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.676214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.676594 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.676595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.676666 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.677225 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.677228 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.678738 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.680336 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.680342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.680343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.680524 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.680980 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.680981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.680981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.681037 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.681470 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.681476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.681518 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.681584 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.681991 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260927 02:56:48.682488 22692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:56:48.687989 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.682825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.0.98.61:32995)
0927 02:56:48.682855 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.682856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.682863 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:48.682880 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.682881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.682888 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.682934 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.683104 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.683577 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.684210 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.684404 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.684452 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.685939 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.685941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.685941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.685946 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.687057 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.687381 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.687382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.687382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.687429 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.687593 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.687593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.687593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.687699 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.687741 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.687879 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.687940 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 02:56:48.688086 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.682689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32995 (local address 127.0.0.1:41688)
0927 02:56:48.682726 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.682734 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.682783 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.682936 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.682936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.683034 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.683059 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.683060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.683632 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.683633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.684135 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.684137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.684454 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.684455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.684501 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.684816 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.684820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.685874 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.687125 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.687126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.687127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.687317 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.687484 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.687485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.687485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.687541 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.687751 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.687751 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.687793 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.687864 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.687997 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":49}
I20260927 02:56:48.688372 22692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:48.688639 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.688531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35900)
0927 02:56:48.688573 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.688618 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:56:48.688756 22643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
I20260927 02:56:48.689114 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.689026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35914)
0927 02:56:48.689066 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.689093 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:56:48.689206 22692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34161: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:56:48.697376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34161
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40009,127.0.98.61:32995,127.0.98.60:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:56:48.725031 22617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } attempt: 1
I20260927 02:56:48.725507 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.725346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35920)
0927 02:56:48.725421 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.725461 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 02:56:48.725858 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.725773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35934)
0927 02:56:48.725814 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.725838 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:56:48.725953 22617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34161: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
I20260927 02:56:48.738415 22692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } attempt: 1
I20260927 02:56:48.738833 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.738700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35940)
0927 02:56:48.738751 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.738806 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:56:48.739117 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.739047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35954)
0927 02:56:48.739078 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.739100 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 02:56:48.739203 22692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34161: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
I20260927 02:56:48.773085 22617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } attempt: 2
I20260927 02:56:48.773528 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.773407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35962)
0927 02:56:48.773464 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.773500 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:56:48.773870 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.773786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35978)
0927 02:56:48.773828 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.773851 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:56:48.773958 22617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34161: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34161: connect: Connection refused (error 111)
W20260927 02:56:48.774300 22699 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:56:48.774479 22699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:48.774510 22699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:48.774544 22699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:48.775696 22699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:48.775733 22699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:48.775753 22699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:48.775768 22699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:48.775791 22699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:48.777240 22699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40009,127.0.98.61:32995,127.0.98.60:34161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:48.777457 22699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:48.777639 22699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:48.781929 22705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:48.781952 22707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:48.781963 22704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:48.782068 22699 server_base.cc:1034] running on GCE node
I20260927 02:56:48.782336 22699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:48.782577 22699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:48.783707 22699 hybrid_clock.cc:648] HybridClock initialized: now 1790477808783709 us; error 42 us; skew 500 ppm
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:48.785919 22699 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:56:48.786324 22699 webserver.cc:460] Webserver started at http://127.0.98.60:43359/ using document root <none> and password file <none>
I20260927 02:56:48.786500 22699 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:48.786536 22699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:48.786624 22699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:48.787575 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/data/instance:
uuid: "36f56515ce26466eaa3c6027540a126d"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "f9954a04a1971b2dc7c9b5b51ce62f70"
server_key_iv: "58dafa1f5a31697d7cb254bdd6ba177f"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.787847 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/wal/instance:
uuid: "36f56515ce26466eaa3c6027540a126d"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "f9954a04a1971b2dc7c9b5b51ce62f70"
server_key_iv: "58dafa1f5a31697d7cb254bdd6ba177f"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.788936 22699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:48.789561 22714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:48.789758 22699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:48.789842 22699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/wal
uuid: "36f56515ce26466eaa3c6027540a126d"
format_stamp: "Formatted at 2026-09-27 02:56:48 on dist-test-slave-mfx9"
server_key: "f9954a04a1971b2dc7c9b5b51ce62f70"
server_key_iv: "58dafa1f5a31697d7cb254bdd6ba177f"
server_key_version: "encryptionkey@0"
I20260927 02:56:48.789906 22699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:48.797374 22699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:48.798879 22699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:48.798991 22699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:48.802460 22699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34161
I20260927 02:56:48.802531 22766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34161 every 8 connection(s)
I20260927 02:56:48.802876 22699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:56:48.803753 22767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:48.805109 22767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:48.807345 22692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } attempt: 2
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:48.811103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 22699
I20260927 02:56:48.811214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477553417353-392-0/minicluster-data/master-2/wal/instance
I20260927 02:56:48.811390 392 external_mini_cluster.cc:1428] Setting key d3bf602e8bbd3107ede39f9f36cc055a
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:56:48.814069 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.805771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.0.98.62:40009)
0927 02:56:48.805839 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.805841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.805945 (+ 104us) server_negotiation.cc:407] Connection header received
0927 02:56:48.806409 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.806410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.806420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.806474 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.807035 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.807547 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.808495 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.808821 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.808890 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.811085 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.811087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.811087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.811096 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.812750 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.813131 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.813132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.813133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.813196 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.813404 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.813405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.813406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.813538 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.813603 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.813898 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.813974 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:56:48.814260 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.805728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40009 (local address 127.0.0.1:55528)
0927 02:56:48.805855 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.805886 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.806311 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.806529 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.806532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.806785 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.806970 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.806974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.807644 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.807645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.808415 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.808417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.808910 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.808911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.809005 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.809469 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.809476 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.810983 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.812836 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.812842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.812844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.813061 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.813258 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.813258 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.813259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.813340 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.813607 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.813612 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.813761 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.813869 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.814045 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260927 02:56:48.814738 22767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:48.815287 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.807564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35984)
0927 02:56:48.807608 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.807618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.807657 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.807919 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.807920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.808047 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.808083 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.808084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.809089 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.809090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.809690 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.809693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.810197 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.810198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.810247 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.810594 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.810599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.811943 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.814216 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.814217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.814217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.814416 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.814634 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.814635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.814635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.814694 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.814995 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.814996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.815041 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.815097 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.815196 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:56:48.815295 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.807632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.0.98.60:34161)
0927 02:56:48.807785 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.807789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.807804 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:56:48.807820 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.807822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.807836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.807896 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.808144 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.809020 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.809759 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.810114 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.810174 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.812021 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.812028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.812028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.812034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.814129 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.814501 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.814503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.814503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.814561 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.814781 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.814782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.814782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.814927 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.814999 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.815134 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.815196 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:56:48.817332 22692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2: Bootstrap starting.
I20260927 02:56:48.817948 22692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:48.818161 22692 log.cc:824] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:48.818820 22692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2: No bootstrap required, opened a new log
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:56:48.820684 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.811698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.98.60:34161)
0927 02:56:48.811811 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.811813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.811955 (+ 142us) server_negotiation.cc:407] Connection header received
0927 02:56:48.812015 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.812016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.812026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.812076 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.812814 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.813392 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.814273 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.814524 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.814565 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.817027 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.817028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.817029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.817035 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.819483 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.819943 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.819944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.819945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.820000 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.820188 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.820189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.820189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.820286 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.820424 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.820493 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.820561 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260927 02:56:48.820959 22692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } }
I20260927 02:56:48.821139 22692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84648d4fc25d4a4fa1aa2a60d28d11b2, State: Initialized, Role: FOLLOWER
I20260927 02:56:48.821326 22692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } }
I20260927 02:56:48.821719 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.821784 22692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:48.822139 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.815262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.0.98.61:32995)
0927 02:56:48.815299 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.815300 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.815678 (+ 378us) server_negotiation.cc:407] Connection header received
0927 02:56:48.815694 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.815695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.815702 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.815741 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.815982 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.816439 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.817105 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.817363 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.817425 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.819332 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.819333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.819333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.819339 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.820785 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.821236 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.821237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.821238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.821293 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.821484 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.821485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.821485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.821611 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.821664 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.822034 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.822088 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:48.822257 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.815177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32995 (local address 127.0.0.1:41692)
0927 02:56:48.815588 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.815597 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.815635 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.815753 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.815754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.815879 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.815908 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.815910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.816510 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.816511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.817034 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.817035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.817423 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.817424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.817485 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.817932 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.817937 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.819241 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.820868 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.820869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.820870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.821052 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.821355 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.821355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.821356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.821418 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.821690 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.821691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.821740 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.821798 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.821877 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397}
I20260927 02:56:48.822598 22767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } has no permanent_uuid. Determining permanent_uuid...
W20260927 02:56:48.823642 22786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:48.823730 22786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:48.823838 22624 master_runner.cc:186] Error getting master registration for 127.0.98.62:40009: OK, instance_id { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" instance_seqno: 1790477808534907 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:56:48.826571 22624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:56:48.821890 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:56:48.829532 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.822817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:35998)
0927 02:56:48.822916 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.822925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.822974 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.823101 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.823101 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.823271 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.823328 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.823330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.824110 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.824111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.824850 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.824852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.825277 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.825278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.825328 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.825652 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.825657 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.827091 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.828624 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.828624 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.828625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.828798 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.828954 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.828955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.828955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.829006 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.829273 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.829274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.829306 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.829353 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.829433 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
I20260927 02:56:48.829825 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.821234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.0.98.62:40009)
0927 02:56:48.821270 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.821272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.821297 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:56:48.821370 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.821371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.821380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.821426 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.821956 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.822607 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.823572 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.824258 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.824302 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.827211 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.827212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.827212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.827218 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.828771 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.829128 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.829129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.829129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.829178 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.829360 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.829361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.829361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.829480 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.829632 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.829674 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.829743 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:56:48.830058 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.822845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.0.98.60:34161)
0927 02:56:48.822924 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.822926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.822962 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:56:48.823021 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.823022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.823030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.823073 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.823402 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.824034 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.824927 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.825187 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.825238 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.827164 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.827165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.827166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.827172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.828541 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.828859 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.828860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.828860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.828900 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.829054 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.829055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.829055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.829200 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.829255 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.829927 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.829994 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 02:56:48.831351 22767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d: Bootstrap starting.
I20260927 02:56:48.832017 22767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:48.832315 22767 log.cc:824] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:48.833040 22767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d: No bootstrap required, opened a new log
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:56:48.835467 22767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } }
I20260927 02:56:48.835596 22767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f56515ce26466eaa3c6027540a126d, State: Initialized, Role: FOLLOWER
I20260927 02:56:48.835669 22767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } }
I20260927 02:56:48.835935 22767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:48.836957 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.830549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.0.98.61:32995)
0927 02:56:48.830580 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.830581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.830856 (+ 275us) server_negotiation.cc:407] Connection header received
0927 02:56:48.830909 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.830910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.830919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.830960 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.831362 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.831822 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.832420 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.832686 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.832748 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.834594 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.834595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.834595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.834602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.835759 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.836033 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.836034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.836034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.836081 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.836257 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.836257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.836258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.836363 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.836491 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.836806 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.836875 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:56:48.837532 22789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
W20260927 02:56:48.837633 22800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:48.837723 22800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:48.838368 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:48.837987 22699 master_runner.cc:186] Error getting master registration for 127.0.98.62:40009: OK, instance_id { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" instance_seqno: 1790477808534907 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:56:48.840065 22699 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:56:48.839357 22591 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4b5afbc772e443ef8648b07ab93c9867"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55542
I20260927 02:56:48.837653 22789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:48.881109 22617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } attempt: 3
Sep 27 02:56:48 dist-test-slave-mfx9 krb5kdc[22536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:56:48.889147 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.881546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.0.98.60:34161)
0927 02:56:48.881649 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.881651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.881659 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:48.881684 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.881685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.881695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.881744 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.882128 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.882920 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.883880 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.884186 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.884261 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.886348 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.886350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.886350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.886358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.888033 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.888400 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.888401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.888402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.888461 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.888652 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.888653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.888653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.888782 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.888841 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.888981 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.889055 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 02:56:48.889159 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.881468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34161 (local address 127.0.0.1:36014)
0927 02:56:48.881527 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:48.881538 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:48.881577 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:48.881779 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:48.881781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:48.881978 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:48.882031 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.882033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.882989 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.882990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.883793 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:48.883796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.884272 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.884273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:48.884329 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.884790 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:48.884795 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:48.886270 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:48.888102 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.888103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.888104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.888328 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.888535 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:48.888536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:48.888536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:48.888598 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:48.888830 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:48.888830 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:48.888888 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 02:56:48.888954 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:56:48.889045 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:56:48.891131 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867: Bootstrap starting.
I20260927 02:56:48.891878 22617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:48.892189 22617 log.cc:824] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:48.892992 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867: No bootstrap required, opened a new log
I20260927 02:56:48.894423 22617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } }
I20260927 02:56:48.894567 22617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5afbc772e443ef8648b07ab93c9867, State: Initialized, Role: FOLLOWER
I20260927 02:56:48.894637 22617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } }
I20260927 02:56:48.894951 22617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:48.894912 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.895102 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:56:48.896732 22814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:48.896835 22814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:48.898695 22549 master_runner.cc:263] No leader master found from master 4b5afbc772e443ef8648b07ab93c9867
I20260927 02:56:48.898738 22549 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:56:48.898864 22591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:56:48.898970 22591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:48.899891 22591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } }
I20260927 02:56:48.900339 22666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b5afbc772e443ef8648b07ab93c9867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2"
I20260927 02:56:48.900472 22666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:48.901185 22666 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5afbc772e443ef8648b07ab93c9867 in term 1.
I20260927 02:56:48.901458 22568 leader_election.cc:302] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5afbc772e443ef8648b07ab93c9867, 84648d4fc25d4a4fa1aa2a60d28d11b2; no voters:
I20260927 02:56:48.901566 22803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:48.901695 22803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [term 1 LEADER]: Becoming Leader. State: Replica: 4b5afbc772e443ef8648b07ab93c9867, State: Running, Role: LEADER
I20260927 02:56:48.901772 22803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } }
I20260927 02:56:48.902158 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b5afbc772e443ef8648b07ab93c9867. Latest consensus state: current_term: 1 leader_uuid: "4b5afbc772e443ef8648b07ab93c9867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.902228 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:48.902859 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b5afbc772e443ef8648b07ab93c9867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "36f56515ce26466eaa3c6027540a126d"
I20260927 02:56:48.902966 22741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:48.903924 22741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5afbc772e443ef8648b07ab93c9867 in term 1.
I20260927 02:56:48.904244 22591 leader_election.cc:288] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [CANDIDATE]: Term 1 election: Requested vote from peers 84648d4fc25d4a4fa1aa2a60d28d11b2 (127.0.98.61:32995), 36f56515ce26466eaa3c6027540a126d (127.0.98.60:34161)
I20260927 02:56:48.904520 22818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:48.904729 22818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:48.906651 22741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [term 1 FOLLOWER]: Refusing update from remote peer 4b5afbc772e443ef8648b07ab93c9867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:48.907027 22803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:48.907477 22666 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [term 1 FOLLOWER]: Refusing update from remote peer 4b5afbc772e443ef8648b07ab93c9867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:48.907802 22803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:48.908902 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b5afbc772e443ef8648b07ab93c9867. Latest consensus state: current_term: 1 leader_uuid: "4b5afbc772e443ef8648b07ab93c9867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.908983 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:48.909927 22789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b5afbc772e443ef8648b07ab93c9867. Latest consensus state: current_term: 1 leader_uuid: "4b5afbc772e443ef8648b07ab93c9867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.910007 22789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:48.911276 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b5afbc772e443ef8648b07ab93c9867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.911352 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:48.911711 22789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b5afbc772e443ef8648b07ab93c9867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.911782 22789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:48.912046 22823 mvcc.cc:205] Tried to move back new op lower bound from 7333797105279520768 to 7333797105262247936. Current Snapshot: MvccSnapshot[applied={T|T < 7333797105279520768}]
I20260927 02:56:48.912428 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b5afbc772e443ef8648b07ab93c9867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.912498 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:48.912840 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b5afbc772e443ef8648b07ab93c9867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5afbc772e443ef8648b07ab93c9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40009 } } peers { permanent_uuid: "84648d4fc25d4a4fa1aa2a60d28d11b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32995 } } peers { permanent_uuid: "36f56515ce26466eaa3c6027540a126d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34161 } } }
I20260927 02:56:48.912914 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:48.913805 22826 mvcc.cc:205] Tried to move back new op lower bound from 7333797105279520768 to 7333797105262247936. Current Snapshot: MvccSnapshot[applied={T|T < 7333797105279520768}]
I20260927 02:56:48.914085 22824 mvcc.cc:205] Tried to move back new op lower bound from 7333797105279520768 to 7333797105262247936. Current Snapshot: MvccSnapshot[applied={T|T < 7333797105279520768}]
I20260927 02:56:48.914604 22818 catalog_manager.cc:1347] Generated new cluster ID: b41467999b54442ba62a32c463105471
I20260927 02:56:48.914649 22818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:48.924678 22818 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:48.925376 22818 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:48.930585 22818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b5afbc772e443ef8648b07ab93c9867: Generated new TSK 0
I20260927 02:56:48.930807 22818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:56:48.932849 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.909587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.98.62:40009)
0927 02:56:48.909626 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.909627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.916819 (+ 7192us) server_negotiation.cc:407] Connection header received
0927 02:56:48.916920 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.916921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.916932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.916960 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.919622 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.920330 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.925460 (+ 5130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.925756 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.925799 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.929924 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.929925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.929926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.929932 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.931538 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.931944 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.931945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.931945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.932010 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.932248 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.932249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.932249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.932374 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.932475 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.932671 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.932767 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:56:48.938050 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.909928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.0.98.61:32995)
0927 02:56:48.909965 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.909967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.917877 (+ 7910us) server_negotiation.cc:407] Connection header received
0927 02:56:48.917967 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.917968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.917977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.918004 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.921376 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.922049 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.924600 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.924894 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.924940 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.931942 (+ 7002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.931944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.931944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.931951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.933476 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.934931 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.934932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.934933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.934990 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.937241 (+ 2251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.937242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.937242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.937377 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.937455 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.937887 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.937963 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:56:48.938495 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:48.910200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.0.98.60:34161)
0927 02:56:48.910234 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.910235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.918798 (+ 8563us) server_negotiation.cc:407] Connection header received
0927 02:56:48.918889 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.918890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.918900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.918927 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.922987 (+ 4060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.923650 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.926968 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:48.927251 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:48.927295 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:48.933991 (+ 6696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:48.933992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:48.933992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:48.933999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:48.935788 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.937607 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.937609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.937609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.937668 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:48.937894 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:48.937895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:48.937895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:48.938018 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:48.938096 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:48.938335 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:56:48.938408 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 02:56:48.963670 22621 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:48.962016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.0.98.62:40009)
0927 02:56:48.962069 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.962071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.963026 (+ 955us) server_negotiation.cc:407] Connection header received
0927 02:56:48.963140 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.963141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.963152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.963181 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.963605 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55566: BlockingRecv error: recv got EOF from 127.0.0.1:55566 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 02:56:48.964401 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:48.962404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.0.98.61:32995)
0927 02:56:48.962442 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.962444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.963868 (+ 1424us) server_negotiation.cc:407] Connection header received
0927 02:56:48.963958 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.963959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.963970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.963996 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.964342 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41710: BlockingRecv error: recv got EOF from 127.0.0.1:41710 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 02:56:48.965221 22772 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:48.962712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.0.98.60:34161)
0927 02:56:48.962750 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.962752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.964746 (+ 1994us) server_negotiation.cc:407] Connection header received
0927 02:56:48.964834 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.964835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.964845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.964872 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.965167 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36034: BlockingRecv error: recv got EOF from 127.0.0.1:36034 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 02:56:48.967720 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:48.966946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.0.98.61:32995)
0927 02:56:48.966981 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.966982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.967244 (+ 262us) server_negotiation.cc:407] Connection header received
0927 02:56:48.967343 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.967344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.967360 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.967387 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.967680 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41720: BlockingRecv error: recv got EOF from 127.0.0.1:41720 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:56:48.970589 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:48.969868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.0.98.61:32995)
0927 02:56:48.969899 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.969900 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.970139 (+ 239us) server_negotiation.cc:407] Connection header received
0927 02:56:48.970230 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.970230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.970240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.970265 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.970550 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41734: BlockingRecv error: recv got EOF from 127.0.0.1:41734 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:56:48.974301 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:48.973607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.0.98.61:32995)
0927 02:56:48.973636 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.973637 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.973873 (+ 236us) server_negotiation.cc:407] Connection header received
0927 02:56:48.973953 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.973954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.973963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.973987 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.974261 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41742: BlockingRecv error: recv got EOF from 127.0.0.1:41742 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 02:56:48.980221 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:48.979509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.0.98.61:32995)
0927 02:56:48.979553 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.979554 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.979789 (+ 235us) server_negotiation.cc:407] Connection header received
0927 02:56:48.979875 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.979876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.979884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.979909 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.980180 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41758: BlockingRecv error: recv got EOF from 127.0.0.1:41758 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 02:56:48.989723 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:48.989212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.0.98.61:32995)
0927 02:56:48.989247 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:48.989248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:48.989332 (+ 84us) server_negotiation.cc:407] Connection header received
0927 02:56:48.989386 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:48.989387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:48.989396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:48.989435 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:48.989668 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41760: BlockingRecv error: recv got EOF from 127.0.0.1:41760 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 02:56:49.007504 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:49.006886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.0.98.61:32995)
0927 02:56:49.006939 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:49.006941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:49.007046 (+ 105us) server_negotiation.cc:407] Connection header received
0927 02:56:49.007115 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:49.007116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:49.007127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:49.007172 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:49.007459 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41768: BlockingRecv error: recv got EOF from 127.0.0.1:41768 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 02:56:49.041220 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:49.040543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.0.98.61:32995)
0927 02:56:49.040596 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:49.040598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:49.040684 (+ 86us) server_negotiation.cc:407] Connection header received
0927 02:56:49.040751 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:49.040752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:49.040762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:49.040810 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:49.041157 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41780: BlockingRecv error: recv got EOF from 127.0.0.1:41780 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 02:56:49.106967 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:49.106247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.0.98.61:32995)
0927 02:56:49.106302 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:49.106304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:49.106384 (+ 80us) server_negotiation.cc:407] Connection header received
0927 02:56:49.106454 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:49.106455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:49.106466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:49.106509 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:49.106904 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41788: BlockingRecv error: recv got EOF from 127.0.0.1:41788 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 02:56:49.236706 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:49.236032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.0.98.61:32995)
0927 02:56:49.236109 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:49.236111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:49.236172 (+ 61us) server_negotiation.cc:407] Connection header received
0927 02:56:49.236258 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:49.236260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:49.236272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:49.236320 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:49.236626 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41800: BlockingRecv error: recv got EOF from 127.0.0.1:41800 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 02:56:49.494429 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:49.493853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.0.98.61:32995)
0927 02:56:49.493911 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:49.493913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:49.493955 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:56:49.494005 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:49.494006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:49.494016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:49.494075 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:49.494370 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41814: BlockingRecv error: recv got EOF from 127.0.0.1:41814 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:56:49.824178 22786 catalog_manager.cc:1259] Loaded cluster ID: b41467999b54442ba62a32c463105471
I20260927 02:56:49.824250 22786 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2: loading cluster ID for follower catalog manager: success
I20260927 02:56:49.825044 22786 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2: acquiring CA information for follower catalog manager: success
I20260927 02:56:49.825331 22786 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84648d4fc25d4a4fa1aa2a60d28d11b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:56:49.838142 22800 catalog_manager.cc:1259] Loaded cluster ID: b41467999b54442ba62a32c463105471
I20260927 02:56:49.838205 22800 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d: loading cluster ID for follower catalog manager: success
I20260927 02:56:49.839023 22800 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d: acquiring CA information for follower catalog manager: success
I20260927 02:56:49.839311 22800 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36f56515ce26466eaa3c6027540a126d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:56:50.010028 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:50.007547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.0.98.61:32995)
0927 02:56:50.007666 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:50.007668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:50.007696 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:56:50.007763 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:50.007764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:50.007831 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:50.007895 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:50.008027 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.008525 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.009268 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.009475 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.009528 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:50.009679 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:50.009809 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:50.009879 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:50.009915 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:56:50.009937 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260927 02:56:50.013787 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:50.011771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.0.98.60:34161)
0927 02:56:50.011855 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:50.011856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:50.011864 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:50.011897 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:50.011898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:50.011929 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:50.011995 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:50.012076 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.012546 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.013203 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.013386 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.013428 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:50.013529 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:50.013620 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:50.013669 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:50.013682 (+ 13us) server_negotiation.cc:299] Negotiation successful
0927 02:56:50.013697 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":40,"threads_started":1}
I20260927 02:56:50.017988 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:50.015446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.0.98.61:32995)
0927 02:56:50.015491 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:50.015493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:50.015607 (+ 114us) server_negotiation.cc:407] Connection header received
0927 02:56:50.015683 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:50.015683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:50.015693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:50.015746 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:50.015909 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.016392 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.017280 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.017569 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.017623 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:50.017777 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:50.017808 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:50.017848 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:50.017897 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:56:50.017918 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:56:50.018393 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:50.015705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.0.98.60:34161)
0927 02:56:50.015742 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:50.015743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:50.015797 (+ 54us) server_negotiation.cc:407] Connection header received
0927 02:56:50.015825 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:50.015827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:50.015836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:50.015894 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:50.015993 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.016701 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.017605 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.017926 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.017982 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:50.018152 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:50.018180 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:50.018203 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:50.018302 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 02:56:50.018327 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:56:50.018584 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:50.015307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.0.98.62:40009)
0927 02:56:50.015415 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:50.015417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:50.015427 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:56:50.015471 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:50.015471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:50.015520 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:50.015610 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:50.015873 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.016661 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.017620 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:50.017928 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:50.017982 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:50.018181 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:50.018286 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:50.018470 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:50.018487 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 02:56:50.018507 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260927 02:56:50.020041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 22549
I20260927 02:56:50.025151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 22624
I20260927 02:56:50.029317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 22699
2026-09-27T02:56:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2656 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:56:50 dist-test-slave-mfx9 krb5kdc[22918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:56:50 dist-test-slave-mfx9 krb5kdc[22918](info): set up 2 sockets
Sep 27 02:56:50 dist-test-slave-mfx9 krb5kdc[22918](info): commencing operation
krb5kdc: starting...
W20260927 02:56:50.974653 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/test-user.keytab.
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:56:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:56:51.080881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36211,127.0.98.61:35893,127.0.98.60:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:56:51.158070 22931 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:56:51.158269 22931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:51.158288 22931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:51.158308 22931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:51.159377 22931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:51.159420 22931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:51.159444 22931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:51.159462 22931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:51.159480 22931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:51.161616 22931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36211,127.0.98.61:35893,127.0.98.60:37557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:51.161881 22931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:51.162135 22931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:51.167501 22936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:51.167582 22937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:51.167497 22939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:51.168196 22931 server_base.cc:1034] running on GCE node
I20260927 02:56:51.168443 22931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:51.168751 22931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:51.169905 22931 hybrid_clock.cc:648] HybridClock initialized: now 1790477811169887 us; error 37 us; skew 500 ppm
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:51.176069 22931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:56:51.176447 22931 webserver.cc:460] Webserver started at http://127.0.98.62:43203/ using document root <none> and password file <none>
I20260927 02:56:51.176637 22931 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:51.176678 22931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:51.176770 22931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:51.177729 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/data/instance:
uuid: "41b1828dfb7d4ff0afddd84c788cd4f2"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "3d495f86720124ffe9d3a8a27b851064"
server_key_iv: "c05757adb6aba0b7c54abc1de458ae08"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.178013 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/wal/instance:
uuid: "41b1828dfb7d4ff0afddd84c788cd4f2"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "3d495f86720124ffe9d3a8a27b851064"
server_key_iv: "c05757adb6aba0b7c54abc1de458ae08"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.179311 22931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:56:51.179975 22946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:51.180114 22931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:56:51.180179 22931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/wal
uuid: "41b1828dfb7d4ff0afddd84c788cd4f2"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "3d495f86720124ffe9d3a8a27b851064"
server_key_iv: "c05757adb6aba0b7c54abc1de458ae08"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.180253 22931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:51.206259 22931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:51.208055 22931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:51.208160 22931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:51.211719 22931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36211
I20260927 02:56:51.211752 22998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36211 every 8 connection(s)
I20260927 02:56:51.212050 22931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:56:51.212973 22999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:51.214402 22999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:51.214854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 22931
I20260927 02:56:51.214928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-0/wal/instance
I20260927 02:56:51.215119 392 external_mini_cluster.cc:1428] Setting key 176375ac582b0ed5c3f9828851af3a4e
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 02:56:51.223385 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.215011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36211 (local address 127.0.0.1:50762)
0927 02:56:51.215189 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.215227 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.215678 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.215925 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.215928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.216240 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.216451 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.216457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.217459 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.217460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.218183 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.218185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.218665 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.218666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.218727 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.219212 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.219220 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.220455 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.222063 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.222069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.222071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.222257 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.222453 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.222453 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.222454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.222528 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.222876 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.222880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.222940 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.223017 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.223232 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":90,"thread_start_us":97,"threads_started":1}
I20260927 02:56:51.223394 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.215027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50762 (local address 127.0.98.62:36211)
0927 02:56:51.215300 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.215304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.215334 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:56:51.215821 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.215823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.215842 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.215916 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.216553 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.217357 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.218267 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.218579 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.218680 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.220530 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.220536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.220537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.220543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.222001 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.222322 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.222324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.222325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.222398 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.222578 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.222579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.222579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.222814 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.222885 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.223044 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.223154 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 02:56:51.224261 22999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:51.224578 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.224457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32770)
0927 02:56:51.224503 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.224552 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:56:51.224713 22948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
I20260927 02:56:51.225063 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.224979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32774)
0927 02:56:51.225016 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.225043 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:56:51.225162 22999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35893: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:56:51.236799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35893
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36211,127.0.98.61:35893,127.0.98.60:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:56:51.274366 22999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } attempt: 1
I20260927 02:56:51.274827 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.274669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32780)
0927 02:56:51.274739 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.274797 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:56:51.275134 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.275055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32796)
0927 02:56:51.275089 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.275111 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:56:51.275229 22999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35893: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
W20260927 02:56:51.314616 23006 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:56:51.314837 23006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:51.314863 23006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:51.314891 23006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:51.316016 23006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:51.316056 23006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:51.316076 23006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:51.316095 23006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:51.316128 23006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:51.317569 23006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36211,127.0.98.61:35893,127.0.98.60:37557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:51.317799 23006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:51.317986 23006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:51.322432 23014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:51.322515 23006 server_base.cc:1034] running on GCE node
W20260927 02:56:51.322434 23012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:51.322474 23011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:51.322990 23006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:51.323277 23006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:51.324436 23006 hybrid_clock.cc:648] HybridClock initialized: now 1790477811324422 us; error 28 us; skew 500 ppm
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:51.326694 23006 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:56:51.327096 23006 webserver.cc:460] Webserver started at http://127.0.98.61:42627/ using document root <none> and password file <none>
I20260927 02:56:51.327279 23006 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:51.327324 23006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:51.327440 23006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:51.328382 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/data/instance:
uuid: "8c7f0ff4ad7040c3bd414236690581f7"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "f9ae0200c94a3b7f1a421ab975206eeb"
server_key_iv: "cf858258fb3409be4599c329b5ef7426"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.328686 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/wal/instance:
uuid: "8c7f0ff4ad7040c3bd414236690581f7"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "f9ae0200c94a3b7f1a421ab975206eeb"
server_key_iv: "cf858258fb3409be4599c329b5ef7426"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.329856 23006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:56:51.330540 23021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:51.330675 23006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:51.330751 23006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/wal
uuid: "8c7f0ff4ad7040c3bd414236690581f7"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "f9ae0200c94a3b7f1a421ab975206eeb"
server_key_iv: "cf858258fb3409be4599c329b5ef7426"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.330842 23006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:51.343364 22999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } attempt: 2
I20260927 02:56:51.343767 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.343651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32806)
0927 02:56:51.343705 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.343736 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:56:51.344123 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.344037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32816)
0927 02:56:51.344074 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.344100 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:56:51.344209 22999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35893: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35893: connect: Connection refused (error 111)
I20260927 02:56:51.344938 23006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:51.346910 23006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:51.347009 23006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:51.350422 23006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35893
I20260927 02:56:51.350449 23073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35893 every 8 connection(s)
I20260927 02:56:51.350733 23006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:56:51.351712 23074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:51.353140 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:51.360563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 23006
I20260927 02:56:51.360669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-1/wal/instance
I20260927 02:56:51.360850 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.353728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.0.98.62:36211)
0927 02:56:51.353794 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.353796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.353944 (+ 148us) server_negotiation.cc:407] Connection header received
0927 02:56:51.354386 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.354387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.354398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.354446 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.355031 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.355485 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.356361 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.356556 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.356603 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.358446 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.358448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.358448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.358454 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.359546 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.359880 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.359881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.359881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.359927 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.360141 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.360142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.360142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.360267 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.360331 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.360705 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.360788 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:56:51.360854 392 external_mini_cluster.cc:1428] Setting key d384282ae3601155306830935f0a44c1
I20260927 02:56:51.361002 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.353699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36211 (local address 127.0.0.1:50770)
0927 02:56:51.353842 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.353876 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.354307 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.354504 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.354507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.354745 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.354959 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.354963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.355549 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.355549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.356294 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.356296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.356623 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.356624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.356737 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.357123 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.357130 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.358365 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.359605 (+ 1240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.359610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.359611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.359825 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.359970 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.359970 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.359970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.360057 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.360329 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.360332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.360504 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.360617 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.360800 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260927 02:56:51.361483 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 02:56:51.368350 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.361711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32820)
0927 02:56:51.361757 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.361905 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.361944 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.362092 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.362092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.362232 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.362282 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.362284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.363095 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.363096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.363815 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.363818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.364251 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.364252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.364323 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.364643 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.364648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.365815 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.367383 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.367384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.367385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.367568 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.367753 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.367754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.367754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.367811 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.368131 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.368131 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.368166 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.368221 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.368293 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:56:51.368451 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.361796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.0.98.61:35893)
0927 02:56:51.361930 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.361935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.361953 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:56:51.361995 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.361997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.362009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.362333 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.362358 (+ 25us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.363026 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.363895 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.364182 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.364236 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.365900 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.365906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.365906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.365912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.367305 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.367636 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.367638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.367638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.367689 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.367920 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.367921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.367921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.368058 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.368141 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.368251 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.368342 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260927 02:56:51.368952 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:51.369239 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.369144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34086)
0927 02:56:51.369176 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.369223 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 02:56:51.369379 23022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
I20260927 02:56:51.369704 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.369629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34088)
0927 02:56:51.369668 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.369689 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:56:51.369793 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:56:51.382773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
/tmp/dist-test-taskoFrgQ1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37557
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36211,127.0.98.61:35893,127.0.98.60:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:56:51.419010 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } attempt: 1
I20260927 02:56:51.419528 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.419382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34098)
0927 02:56:51.419449 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.419494 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:56:51.419916 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.419825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34114)
0927 02:56:51.419867 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.419896 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:56:51.420007 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
I20260927 02:56:51.435366 22999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } attempt: 3
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:56:51.442186 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.435657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32828)
0927 02:56:51.435729 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.435742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.435802 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.435941 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.435942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.436115 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.436188 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.436190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.436824 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.436826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.437659 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.437663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.438005 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.438006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.438083 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.438478 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.438484 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.439787 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.441169 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.441170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.441171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.441336 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.441504 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.441505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.441506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.441574 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.441820 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.441822 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.441905 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.441992 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.442086 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:56:51.442543 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.435733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.0.98.61:35893)
0927 02:56:51.435794 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.435796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.435806 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:56:51.435864 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.435865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.435875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.435923 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.436252 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.436740 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.437741 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.437943 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.437990 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.439865 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.439866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.439867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.439873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.441093 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.441394 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.441395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.441396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.441443 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.441631 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.441632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.441633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.441752 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.441810 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.442376 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.442460 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:56:51.443032 22999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:56:51.443325 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.443245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34128)
0927 02:56:51.443277 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.443304 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 02:56:51.443583 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.443521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34130)
0927 02:56:51.443549 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.443569 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 02:56:51.443670 22999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
W20260927 02:56:51.461668 23081 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:56:51.461846 23081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:56:51.461870 23081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:56:51.461895 23081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:56:51.463009 23081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:56:51.463047 23081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:56:51.463064 23081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:56:51.463079 23081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:56:51.463095 23081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:56:51.464495 23081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36211,127.0.98.61:35893,127.0.98.60:37557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:56:51.464725 23081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:56:51.464911 23081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:56:51.469215 23087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:51.469233 23086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:56:51.469257 23089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:56:51.469506 23081 server_base.cc:1034] running on GCE node
I20260927 02:56:51.469723 23081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:56:51.469964 23081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:56:51.471096 23081 hybrid_clock.cc:648] HybridClock initialized: now 1790477811471082 us; error 28 us; skew 500 ppm
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:56:51.473315 23081 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:56:51.473740 23081 webserver.cc:460] Webserver started at http://127.0.98.60:38725/ using document root <none> and password file <none>
I20260927 02:56:51.473933 23081 fs_manager.cc:359] Metadata directory not provided
I20260927 02:56:51.473977 23081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:56:51.474123 23081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:56:51.474831 22999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } attempt: 1
I20260927 02:56:51.475205 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.475086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34136)
0927 02:56:51.475144 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.475177 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:56:51.475189 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/data/instance:
uuid: "c8964a2076224dbc8ad55a7082c2ff11"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "e608e207d9f9719dbbc301158fb51150"
server_key_iv: "5196bc64bef134d67f5b29805c1a572d"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.475526 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/wal/instance:
uuid: "c8964a2076224dbc8ad55a7082c2ff11"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "e608e207d9f9719dbbc301158fb51150"
server_key_iv: "5196bc64bef134d67f5b29805c1a572d"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.475554 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.475453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34152)
0927 02:56:51.475496 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.475524 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:56:51.475644 22999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
I20260927 02:56:51.476729 23081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:56:51.477396 23096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:56:51.477550 23081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:56:51.477619 23081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/wal
uuid: "c8964a2076224dbc8ad55a7082c2ff11"
format_stamp: "Formatted at 2026-09-27 02:56:51 on dist-test-slave-mfx9"
server_key: "e608e207d9f9719dbbc301158fb51150"
server_key_iv: "5196bc64bef134d67f5b29805c1a572d"
server_key_version: "encryptionkey@0"
I20260927 02:56:51.477682 23081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:56:51.488161 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } attempt: 2
I20260927 02:56:51.488535 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.488428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34162)
0927 02:56:51.488478 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.488508 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:56:51.488886 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.488806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34166)
0927 02:56:51.488845 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.488870 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:56:51.488960 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37557: connect: Connection refused (error 111)
I20260927 02:56:51.502615 23081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:56:51.504527 23081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:56:51.504634 23081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:56:51.508394 23081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37557
I20260927 02:56:51.508428 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37557 every 8 connection(s)
I20260927 02:56:51.508757 23081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:56:51.509655 23149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:56:51.511197 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:56:51.516804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu as pid 23081
I20260927 02:56:51.516907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoFrgQ1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477553417353-392-0/minicluster-data/master-2/wal/instance
I20260927 02:56:51.517076 392 external_mini_cluster.cc:1428] Setting key cc22c82df3d35bb791e92b3fa59f3b7a
I20260927 02:56:51.519623 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.511841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.0.98.62:36211)
0927 02:56:51.511893 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.511895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.512103 (+ 208us) server_negotiation.cc:407] Connection header received
0927 02:56:51.512516 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.512517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.512527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.512586 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.513186 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.513678 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.514588 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.514850 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.514903 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.516858 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.516860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.516860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.516868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.518270 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.518629 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.518630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.518631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.518691 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.519010 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.519011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.519011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.519147 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.519206 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.519494 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.519559 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:56:51.519820 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.511809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36211 (local address 127.0.0.1:50782)
0927 02:56:51.511956 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.511997 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.512436 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.512650 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.512653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.512918 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.513106 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.513110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.513758 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.513759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.514511 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.514513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.514937 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.514938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.515034 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.515442 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.515450 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.516767 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.518346 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.518353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.518355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.518564 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.518812 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.518813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.518814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.518926 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.519199 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.519203 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.519332 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.519443 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.519590 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260927 02:56:51.520450 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:56:51.525513 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.517430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.0.98.60:37557)
0927 02:56:51.517574 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.517579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.517604 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:56:51.517701 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.517704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.517715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.517778 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.518339 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.519258 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.520177 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.520581 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.520642 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.522877 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.522884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.522884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.522889 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.524450 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.524776 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.524780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.524780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.524839 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.525013 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.525014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.525014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.525167 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.525296 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.525384 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.525441 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260927 02:56:51.526851 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.520664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35893 (local address 127.0.0.1:32830)
0927 02:56:51.520756 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.520765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.520802 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.520920 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.520921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.521039 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.521078 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.521079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.521645 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.521645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.522324 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.522326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.522732 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.522733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.522826 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.523175 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.523179 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.524349 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.525915 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.525915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.525916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.526083 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.526252 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.526253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.526253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.526314 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.526544 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.526545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.526587 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.526658 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.526745 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 02:56:51.526863 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.520732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.0.98.61:35893)
0927 02:56:51.520785 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.520787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.520794 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:51.520851 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.520852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.520860 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.520907 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.521143 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.521582 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.522414 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.522660 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.522710 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.524428 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.524430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.524430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.524436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.525850 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.526150 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.526151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.526151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.526200 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.526362 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.526363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.526363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.526487 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.526535 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.526678 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.526738 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:56:51.527146 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:56:51.532857 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.526260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.0.98.62:36211)
0927 02:56:51.526299 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.526300 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.526308 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:51.526329 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.526330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.526339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.526387 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.526865 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.527475 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.528333 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.528539 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.528600 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.530616 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.530618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.530618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.530625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.531928 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.532211 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.532212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.532212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.532262 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.532415 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.532416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.532416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.532514 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.532662 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.532696 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.532769 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:56:51.533075 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.527382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34182)
0927 02:56:51.527418 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.527426 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.527456 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.527567 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.527568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.527670 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.527695 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.527696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.528468 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.528469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.529098 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.529100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.529536 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.529536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.529571 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.529871 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.529876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.530886 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.532246 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.532247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.532247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.532391 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.532542 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.532543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.532543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.532590 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.532779 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.532780 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.532814 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.532877 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.532970 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:56:51.533145 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.527428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.0.98.60:37557)
0927 02:56:51.527456 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.527458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.527464 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:56:51.527497 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.527498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.527506 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.527553 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.527762 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.528415 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.529169 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.529468 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.529539 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.530955 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.530956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.530957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.530963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.532174 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.532444 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.532445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.532445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.532491 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.532637 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.532638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.532638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.532730 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.532779 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.532909 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.533025 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":15}
I20260927 02:56:51.534658 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11: Bootstrap starting.
I20260927 02:56:51.535516 23149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:51.535813 23149 log.cc:824] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:51.536585 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11: No bootstrap required, opened a new log
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:56:51.538394 23149 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } }
I20260927 02:56:51.538558 23149 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8964a2076224dbc8ad55a7082c2ff11, State: Initialized, Role: FOLLOWER
I20260927 02:56:51.538666 23149 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } }
I20260927 02:56:51.539017 23149 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:51.538991 23155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.539078 23155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:56:51.540660 23166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 02:56:51.540690 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.533726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.0.98.61:35893)
0927 02:56:51.533764 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.533766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.533774 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:51.533796 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.533797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.533806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.533866 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.534261 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.534917 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.535511 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.535785 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.535854 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.537996 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.537998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.537998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.538006 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.539617 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.539926 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.539927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.539928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.539988 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.540216 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.540217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.540218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.540341 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.540498 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.540537 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.540605 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:56:51.540982 23166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:51.541899 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:56:51.542847 22973 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784
I20260927 02:56:51.541780 23081 master_runner.cc:186] Error getting master registration for 127.0.98.62:36211: OK, instance_id { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" instance_seqno: 1790477811210693 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:56:51.544139 23081 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:56:51.550817 22999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } attempt: 2
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:56:51.557332 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.551095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34184)
0927 02:56:51.551167 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.551181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.551248 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.551368 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.551369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.551529 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.551568 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.551569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.552151 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.552152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.552634 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.552636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.552929 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.552930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.552982 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.553419 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.553424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.554705 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.556319 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.556320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.556321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.556530 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.556698 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.556699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.556699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.556756 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.557028 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.557029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.557074 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.557146 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.557239 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 02:56:51.557335 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.551155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.0.98.60:37557)
0927 02:56:51.551193 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.551195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.551231 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:56:51.551302 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.551302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.551312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.551358 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.551629 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.552085 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.552699 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.552875 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.552926 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.554806 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.554808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.554808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.554815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.556244 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.556594 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.556594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.556595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.556645 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.556813 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.556814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.556814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.556971 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.557031 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.557170 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.557246 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:56:51.558885 22999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2: Bootstrap starting.
I20260927 02:56:51.559593 22999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:51.559886 22999 log.cc:824] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:51.560642 22999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2: No bootstrap required, opened a new log
I20260927 02:56:51.561934 22999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } }
I20260927 02:56:51.562072 22999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b1828dfb7d4ff0afddd84c788cd4f2, State: Initialized, Role: FOLLOWER
I20260927 02:56:51.562161 22999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } }
I20260927 02:56:51.562467 22999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:51.562423 23169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.562536 23169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:51.562567 22973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:56:51.562616 22973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:51.563303 22973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } }
I20260927 02:56:51.563802 22973 leader_election.cc:288] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8c7f0ff4ad7040c3bd414236690581f7 (127.0.98.61:35893), c8964a2076224dbc8ad55a7082c2ff11 (127.0.98.60:37557)
I20260927 02:56:51.563964 23123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c8964a2076224dbc8ad55a7082c2ff11"
I20260927 02:56:51.564092 23123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [term 0 FOLLOWER]: Advancing to term 1
W20260927 02:56:51.564565 23180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:56:51.564670 23180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:56:51.564947 23123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b1828dfb7d4ff0afddd84c788cd4f2 in term 1.
I20260927 02:56:51.565508 22947 leader_election.cc:302] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41b1828dfb7d4ff0afddd84c788cd4f2, c8964a2076224dbc8ad55a7082c2ff11; no voters:
I20260927 02:56:51.565659 23169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:56:51.565785 23169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [term 1 LEADER]: Becoming Leader. State: Replica: 41b1828dfb7d4ff0afddd84c788cd4f2, State: Running, Role: LEADER
I20260927 02:56:51.565869 23169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } }
I20260927 02:56:51.566249 23169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41b1828dfb7d4ff0afddd84c788cd4f2. Latest consensus state: current_term: 1 leader_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.566320 23169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:51.567687 22931 master_runner.cc:186] Error getting master registration for 127.0.98.61:35893: OK, instance_id { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" instance_seqno: 1790477811349465 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:56:51.569825 22931 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:56:51.574265 23191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:56:51.575531 23191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:56:51.577225 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.565632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.0.98.62:36211)
0927 02:56:51.566441 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.566442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.567493 (+ 1051us) server_negotiation.cc:407] Connection header received
0927 02:56:51.567633 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.567634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.567660 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.567714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.568750 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.569492 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.571315 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.571602 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.571640 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.573565 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.573566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.573566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.573573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.575685 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.576223 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.576224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.576225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.576281 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.576549 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.576550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.576550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.576683 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.576755 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.577050 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.577126 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791}
I20260927 02:56:51.577510 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.565024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.0.98.61:35893)
0927 02:56:51.565066 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.565068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.566143 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 02:56:51.566285 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.566286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.566309 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.566340 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.567658 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.568351 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.569360 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.569634 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.569679 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.573391 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.573392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.573393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.573401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.574949 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.576654 (+ 1705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.576655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.576655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.576720 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.576933 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.576934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.576934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.577061 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.577128 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.577349 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.577429 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:56:51.580081 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } attempt: 3
I20260927 02:56:51.581332 23123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [term 1 FOLLOWER]: Refusing update from remote peer 41b1828dfb7d4ff0afddd84c788cd4f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:51.581661 23188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:51.583004 23155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41b1828dfb7d4ff0afddd84c788cd4f2. Latest consensus state: current_term: 1 leader_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.583096 23155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:51.583535 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.565822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.0.98.60:37557)
0927 02:56:51.565855 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.565857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.570081 (+ 4224us) server_negotiation.cc:407] Connection header received
0927 02:56:51.570173 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.570174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.570184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.570223 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.571724 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.572423 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.577674 (+ 5251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.577959 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.578002 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.580101 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.580102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.580103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.580110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.581906 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.582471 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.582472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.582473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.582540 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.582839 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.582840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.582841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.582980 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.583055 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.583370 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.583445 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:56:51.585775 23188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.585876 23188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:51.586236 23155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.586315 23155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:51.586625 23192 mvcc.cc:205] Tried to move back new op lower bound from 7333797116234334208 to 7333797116174405632. Current Snapshot: MvccSnapshot[applied={T|T < 7333797116234334208}]
Sep 27 02:56:51 dist-test-slave-mfx9 krb5kdc[22918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:56:51.588464 23194 mvcc.cc:205] Tried to move back new op lower bound from 7333797116234334208 to 7333797116174405632. Current Snapshot: MvccSnapshot[applied={T|T < 7333797116234334208}]
I20260927 02:56:51.591666 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.580310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37557 (local address 127.0.0.1:34212)
0927 02:56:51.580409 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:56:51.580419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:56:51.580451 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:56:51.583707 (+ 3256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:56:51.583708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:56:51.583867 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:56:51.583913 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.583915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.584812 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.584813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.585607 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:56:51.585609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.586175 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.586176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:56:51.586247 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.586686 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:56:51.586703 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:56:51.588148 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:56:51.590538 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.590540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.590540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.590746 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.591043 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:56:51.591044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:56:51.591045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:56:51.591109 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:56:51.591389 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:56:51.591390 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:56:51.591433 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:56:51.591478 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:56:51.591571 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 02:56:51.591961 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:51.580536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.0.98.60:37557)
0927 02:56:51.583600 (+ 3064us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.583602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.583608 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:56:51.583624 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.583625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.583634 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.583695 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.584007 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.584740 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.585734 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:51.586041 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:51.586095 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:51.588222 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:56:51.588223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:56:51.588223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:56:51.588230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:56:51.590421 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.590876 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.590877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.590877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.590958 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:56:51.591177 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:56:51.591178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:56:51.591178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:56:51.591316 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:56:51.591777 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:51.591823 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:56:51.591887 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3051,"thread_start_us":90,"threads_started":1}
I20260927 02:56:51.592067 23191 catalog_manager.cc:1347] Generated new cluster ID: 2f2b23b7376049c9b610f352c2e1df6e
I20260927 02:56:51.592121 23191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:56:51.597576 23074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7: Bootstrap starting.
I20260927 02:56:51.598297 23074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7: Neither blocks nor log segments found. Creating new log.
I20260927 02:56:51.598578 23074 log.cc:824] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:56:51.599766 23191 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:56:51.600456 23074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7: No bootstrap required, opened a new log
I20260927 02:56:51.600466 23191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:56:51.602399 23074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } }
I20260927 02:56:51.602535 23074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7f0ff4ad7040c3bd414236690581f7, State: Initialized, Role: FOLLOWER
I20260927 02:56:51.602614 23074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } }
I20260927 02:56:51.602938 23074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:56:51.602967 23048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8c7f0ff4ad7040c3bd414236690581f7"
I20260927 02:56:51.603067 23048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:56:51.603811 23048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b1828dfb7d4ff0afddd84c788cd4f2 in term 1.
I20260927 02:56:51.604215 23047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [term 1 FOLLOWER]: Refusing update from remote peer 41b1828dfb7d4ff0afddd84c788cd4f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:56:51.604808 23188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:56:51.607100 23210 catalog_manager.cc:1259] Loaded cluster ID: 2f2b23b7376049c9b610f352c2e1df6e
I20260927 02:56:51.607137 23210 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7: loading cluster ID for follower catalog manager: success
I20260927 02:56:51.608201 23210 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7: acquiring CA information for follower catalog manager: success
W20260927 02:56:51.608317 23210 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260927 02:56:51.608510 23199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.608585 23199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:51.608628 23199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41b1828dfb7d4ff0afddd84c788cd4f2. Latest consensus state: current_term: 1 leader_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.608664 23199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:51.608688 23199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.608721 23199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:56:51.611060 23188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1828dfb7d4ff0afddd84c788cd4f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } peers { permanent_uuid: "8c7f0ff4ad7040c3bd414236690581f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35893 } } peers { permanent_uuid: "c8964a2076224dbc8ad55a7082c2ff11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37557 } } }
I20260927 02:56:51.611141 23188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:56:51.614830 23191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41b1828dfb7d4ff0afddd84c788cd4f2: Generated new TSK 0
I20260927 02:56:51.615044 23191 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:56:51.633868 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.632200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.98.61:35893)
0927 02:56:51.632248 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.632251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.633121 (+ 870us) server_negotiation.cc:407] Connection header received
0927 02:56:51.633235 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.633236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.633247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.633291 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.633786 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32842: BlockingRecv error: recv got EOF from 127.0.0.1:32842 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 02:56:51.634626 23002 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.632588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.0.98.62:36211)
0927 02:56:51.632625 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.632626 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.634101 (+ 1475us) server_negotiation.cc:407] Connection header received
0927 02:56:51.634209 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.634210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.634220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.634249 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.634564 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50790: BlockingRecv error: recv got EOF from 127.0.0.1:50790 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:56:51.635345 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.632878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.0.98.60:37557)
0927 02:56:51.632916 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.632918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.634869 (+ 1951us) server_negotiation.cc:407] Connection header received
0927 02:56:51.634960 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.634961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.634970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.634996 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.635289 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34218: BlockingRecv error: recv got EOF from 127.0.0.1:34218 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 02:56:51.643437 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.639953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.0.98.61:35893)
0927 02:56:51.639986 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.639988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.642913 (+ 2925us) server_negotiation.cc:407] Connection header received
0927 02:56:51.643020 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.643021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.643039 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.643068 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.643391 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32852: BlockingRecv error: recv got EOF from 127.0.0.1:32852 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:56:51.646349 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.645607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.0.98.61:35893)
0927 02:56:51.645638 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.645640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.645867 (+ 227us) server_negotiation.cc:407] Connection header received
0927 02:56:51.645957 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.645958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.645968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.645995 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.646305 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32862: BlockingRecv error: recv got EOF from 127.0.0.1:32862 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:56:51.650113 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.649404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.0.98.61:35893)
0927 02:56:51.649435 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.649436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.649661 (+ 225us) server_negotiation.cc:407] Connection header received
0927 02:56:51.649752 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.649752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.649762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.649788 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.650071 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32870: BlockingRecv error: recv got EOF from 127.0.0.1:32870 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 02:56:51.655961 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.655223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.0.98.61:35893)
0927 02:56:51.655254 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.655256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.655477 (+ 221us) server_negotiation.cc:407] Connection header received
0927 02:56:51.655569 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.655570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.655579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.655608 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.655914 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32880: BlockingRecv error: recv got EOF from 127.0.0.1:32880 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:56:51.665768 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.665036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.0.98.61:35893)
0927 02:56:51.665070 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.665072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.665296 (+ 224us) server_negotiation.cc:407] Connection header received
0927 02:56:51.665386 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.665387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.665397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.665423 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.665723 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32888: BlockingRecv error: recv got EOF from 127.0.0.1:32888 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:56:51.684090 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.683167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.0.98.61:35893)
0927 02:56:51.683211 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.683214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.683484 (+ 270us) server_negotiation.cc:407] Connection header received
0927 02:56:51.683584 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.683585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.683597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.683625 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.684031 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32898: BlockingRecv error: recv got EOF from 127.0.0.1:32898 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:56:51.718295 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.717471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.0.98.61:35893)
0927 02:56:51.717519 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.717521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.717801 (+ 280us) server_negotiation.cc:407] Connection header received
0927 02:56:51.717860 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.717861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.717873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.717922 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.718234 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32904: BlockingRecv error: recv got EOF from 127.0.0.1:32904 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:56:51.783946 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.783274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.0.98.61:35893)
0927 02:56:51.783345 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.783347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.783475 (+ 128us) server_negotiation.cc:407] Connection header received
0927 02:56:51.783535 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.783536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.783547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.783591 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.783887 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32910: BlockingRecv error: recv got EOF from 127.0.0.1:32910 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 02:56:51.913599 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:51.913025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.0.98.61:35893)
0927 02:56:51.913089 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:51.913091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:51.913159 (+ 68us) server_negotiation.cc:407] Connection header received
0927 02:56:51.913207 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:51.913208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:51.913218 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:51.913278 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:51.913551 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32912: BlockingRecv error: recv got EOF from 127.0.0.1:32912 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:56:52.171310 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:56:52.170727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.0.98.61:35893)
0927 02:56:52.170791 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:52.170793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:52.170865 (+ 72us) server_negotiation.cc:407] Connection header received
0927 02:56:52.170919 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:52.170920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:52.170931 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:52.170976 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:56:52.171252 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32914: BlockingRecv error: recv got EOF from 127.0.0.1:32914 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:56:52.541436 23166 catalog_manager.cc:1259] Loaded cluster ID: 2f2b23b7376049c9b610f352c2e1df6e
I20260927 02:56:52.541507 23166 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11: loading cluster ID for follower catalog manager: success
I20260927 02:56:52.542335 23166 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11: acquiring CA information for follower catalog manager: success
I20260927 02:56:52.542583 23166 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8964a2076224dbc8ad55a7082c2ff11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:56:52.608841 23210 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c7f0ff4ad7040c3bd414236690581f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:56:52.686655 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:52.684406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.98.61:35893)
0927 02:56:52.684505 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:52.684507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:52.684515 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:56:52.684592 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:52.684593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:52.684641 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:52.684714 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:52.684830 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.685319 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.686042 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.686232 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.686274 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:52.686385 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:52.686488 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:52.686539 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:52.686554 (+ 15us) server_negotiation.cc:299] Negotiation successful
0927 02:56:52.686571 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260927 02:56:52.690388 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:52.688283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.0.98.60:37557)
0927 02:56:52.688368 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:52.688370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:52.688377 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:56:52.688409 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:52.688410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:52.688442 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:52.688504 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:52.688582 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.689043 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.689682 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.689876 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.689925 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:52.690111 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:52.690208 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:52.690263 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:52.690297 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 02:56:52.690312 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":45,"threads_started":1}
I20260927 02:56:52.694523 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:52.692196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.0.98.60:37557)
0927 02:56:52.692229 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:52.692231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:52.692237 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:56:52.692255 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:52.692256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:52.692264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:52.692316 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:52.692479 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.692941 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.693802 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.694074 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.694135 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:52.694314 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:52.694344 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:52.694389 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:52.694437 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:56:52.694456 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:56:52.694805 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:52.691965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.0.98.61:35893)
0927 02:56:52.692003 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:52.692005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:52.692167 (+ 162us) server_negotiation.cc:407] Connection header received
0927 02:56:52.692275 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:52.692276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:52.692287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:52.692334 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:52.692480 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.693202 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.694061 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.694292 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.694349 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:52.694494 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:52.694526 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:52.694558 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:52.694688 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 02:56:52.694712 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:56:52.695169 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:56:52.691920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.62:36211)
0927 02:56:52.692058 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:56:52.692060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:56:52.692071 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:56:52.692168 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:56:52.692169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:56:52.692359 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:56:52.692428 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:56:52.692540 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.693282 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.694133 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:56:52.694411 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:56:52.694469 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:56:52.694677 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:56:52.694796 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:56:52.694864 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:56:52.695061 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:56:52.695083 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":45,"threads_started":1}
I20260927 02:56:52.696767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 22931
I20260927 02:56:52.701797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 23006
I20260927 02:56:52.706070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoFrgQ1/build/release/bin/kudu with pid 23081
2026-09-27T02:56:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2676 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5333 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (259294 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST